A multi-sensor approach for grasping and 3D interaction

2006

Interactive virtual characters are nowadays commonplace in games, animations, and Virtual Reality (VR) applications. However, relatively few work has so far considered the animation of interactive object manipulations performed by virtual humans. In this paper, we first present a hierarchical control architecture incorporating plans, behaviors, and motor programs that enables virtual humans to accurately manipulate scene objects using different grasp types. Furthermore, as second main contribution, we introduce a method by which virtual humans learn to imitate object manipulations performed by human VR users. To this end, movements of the VR user are analyzed and processed into abstract actions. A new data structure called grasp events is used for storing information about user interactions with scene objects. High-level plans are instantiated based on grasp events to drive the virtual humans' animation. Due to their high-level representation, recorded manipulations often naturally adapt to new situations without losing plausibility.

2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2006

This paper presents a control approach for human grasp simulation in a real-time context. We consider a 12-dof hand grasping quasi-rigid objects, either for fine manipulation or perturbation rejection. The objective is to translate and rotate the object with respect to non-sliding conditions. In order to improve the visual realism and the stability of the simulation, we use a locally deformable model on fingertips with friction. The resolution of contact forces is used for controlling the system with a set of 23 muscles. This paper presents also a new technique for solving the grasping force optimization problem, allowing real time simulation. This work is part of a virtual human project and combines robotic control experiments and theoretical laws of deformations in a global simulation.

Robotics and Autonomous Systems, 2011

This overview presents computational algorithms for generating 3D object grasps with autonomous multi-fingered robotic hands. Robotic grasping has been an active research subject for decades, and a great deal of effort has been spent on grasp synthesis algorithms. Existing papers focus on reviewing the mechanics of grasping and the finger-object contact interactions Bicchi and Kumar [12] or robot hand design and their control Al-Gallaf et al. (1993) [70]. Robot grasp synthesis algorithms have been reviewed in [71], but since then an important progress has been made toward applying learning techniques to the grasping problem. This overview focuses on analytical as well as empirical grasp synthesis approaches.

Volume 2: 29th Computers and Information in Engineering Conference, Parts A and B, 2009

This paper compares two approaches to controlling virtual hands in grasping simulation, and it investigates the ability of humans to control with their hands a virtual hand model in manipulative tasks. In our setup, the users are following their interaction in a desktop virtual reality environment, in order to determine the feasibility of user studies for user -virtual product interaction. Because no real force feed-back is provided, the user can decide upon the correctness of the grasping posture only from visual feedback. The hand and the virtual object interaction are computed from a simulation, therefore accurate spatial position of the real hand and fingers are needed to be measured in real time. Our simulation program is using the Nvidia PhysX SDK. We have implemented two control mechanisms, which enables the users of the system to manipulate the virtual hand. The first mechanism controls the motion of the virtual hand and grasping forces based on the principles of kinematics and energy transfer by contact simulation. The second mechanism relies on the principles of multibody dynamics, controlling the motion of the hand by PD controllers, and applying joint torques to the hand in order to exert forces on the grasped object. In this paper, we compare how well these principles perform in (a) accurately moving the virtual hand in the simulation space, (b) accurately positioning the fingers on the grasped objects, and (c) controlling the grasping forces on the objects.

IAES International Journal of Robotics and Automation (IJRA), 2023

Human hands are essential in everyday tasks, mainly manipulating and grasping objects. Thus, accurate and precise three-dimensional (3D) models of digitally reconstructed hands are valuable to the world of ergonomics. A 3D scan-to-render system called the “3D hands model rendering using a 6-degrees of freedom (DoF) collaborative robot” is proposed to ensure that a person receives the best possible outcome for their unique anatomy. The description implies this is using a 6-DoF robot with a two-dimensional (2D) camera sensor that will encompass all forms of the production line in a timely, low-cost, precise, and accurate manner so that an individual can go to and scan their hand and have an actual 3D reconstruction print within the same facility, the same day. It is expected to generate an accurate hand model using structure from motion (SFM) system techniques to create a dense point cloud using photogrammetry. The point cloud is used to develop the tetrahedral mesh of the surface of the hand. This mesh is then refined to filter out the noise of the point cloud. The mesh can produce a precise 3D model that can tailor products to the consumer's needs. The results show the effectiveness of the 3D model of the hand.

Presence (Cambridge, Mass.), 1996

Simulating a human figure performing a manual task requires that the agent interact with objects in the environment in a realistic manner. Graphic or programming interfaces to control human figure animation, however, do not allow the animator to instruct the system with concise "high-level" commands. Instructions coming from a high-level planner cannot be directly given to a synthetic agent because they do not specify such details as which end-effector to use or where on the object to grasp. Because current animation systems require joint angle displacement descriptions of motion--even for motions that incorporate upwards of 15 joints--an efficient connection between high-level specifications and low-level hand joint motion is required. In this paper we describe a system that directs task-level, general-purpose, object grasping for a simulated human agent. The Object-Specific Reasoner (OSR) is a reasoning module that uses knowledge of the object of the underspecified actio...

Computers & Graphics, 2019

Interaction in virtual reality (VR) environments is essential to achieve a pleasant and immersive experience. Most of the currently existing VR applications, lack of robust object grasping and manipulation, which are the cornerstone of interactive systems. Therefore, we propose a realistic, flexible and robust grasping system that enables rich and real-time interactions in virtual environments. It is visually realistic because it is completely user-controlled, flexible because it can be used for different hand configurations, and robust because it allows the manipulation of objects regardless their geometry, i.e. hand is automatically fitted to the object shape. In order to validate our proposal, an exhaustive qualitative and quantitative performance analysis has been carried out. On the one hand, qualitative evaluation was used in the assessment of the abstract aspects such as: hand movement realism, interaction realism and motor control. On the other hand, for the quantitative evaluation a novel error metric has been proposed to visually analyze the performed grips. This metric is based on the computation of the distance from the finger phalanges to the nearest contact point on the object surface. These contact points can be used with different application purposes, mainly in the field of robotics. As a conclusion, system evaluation reports a similar performance between users with previous experience in virtual reality applications and inexperienced users, referring to a steep learning curve.

Computer Graphics Forum, 2003

We present new techniques that use motion planning algorithms based on probabilistic roadmaps to control 22 degrees of freedom (DOFs) of human-like characters in interactive applications. Our main purpose is the automatic synthesis of collision-free reaching motions for both arms, with automatic column control and leg flexion. Generated motions are collision-free, in equilibrium, and respect articulation range limits. In order to deal with the high (22) dimension of our configuration space, we bias the random distribution of configurations to favor postures most useful for reaching and grasping. In addition, extensions are presented in order to interactively generate object manipulation sequences: a probabilistic inverse kinematics solver for proposing goal postures matching pre-designed grasps; dynamic update of roadmaps when obstacles change position; online planning of object location transfer; and an automatic stepping control to enlarge the character's reachable space. This is, to our knowledge, the first time probabilistic planning techniques are used to automatically generate collision-free reaching motions involving the entire body of a human-like character at interactive frame rates. Categories and Subject Descriptors (according to ACM CCS): I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism † Work done while at EPFL -Virtual Reality Lab nity. Most of the techniques developed 1 have not sufficiently explored this domain.

Rapid simulation of object grasp is an important component of ergonomic analysis with digital human figure models. A large number of research papers have been presented on grasp simulation, but grasp simulation in commercial ergonomics tools has changed little in the past 10 years, in part because most research approaches to grasp simulation are difficult to implement for general-purpose simulation. This paper presents the development of a data-based, kinematic grasp simulation method that is integrated into a whole-body motion simulation framework. The right-hand motions of six subjects were recorded using a CyberGlove as they reached toward and grasped seven generic objects using five grasp types. A principal component analysis of 22 joint angle trajectories defining the hand pose motion was conducted. A regression model was developed to predict the hand motion for each of the grasp types using the object dimension scaled by hand length. The resulting model is integrated into the Human Motion Simulation Framework, which includes hand trajectory prediction and uses collision detection to fine-tune the grasp execution and terminal posture. The integration includes several novel elements, including an automatically positioned virtual end-effector and user configurability of key grasp attributes while preserving the smooth motion generation provided by the data-based model. An automotive case study using the new model demonstrated a significant reduction in the time required to produce a visually reasonable simulation.

The analysis and the understanding of object manipulation scenarios based on computer vision techniques can be greatly facilitated if we can gain access to the full articulation of the manipulating hands and the 3D pose of the manipulated objects. Currently, there exist methods for tracking hands in interaction with objects whose 3D models are known. There are also methods that can reconstruct 3D models of objects that are partially observable in each frame of a sequence. However, to the best of our knowledge, no method can track hands in interaction with unknown objects. In this paper we propose such a method. Experimental results show that hand tracking can be achieved with an accuracy that is comparable to the one obtained by methods that assume knowledge of the object models. Additionally, as a by-product, the proposed method delivers accurate 3D models of the manipulated objects.