Beyond Teleoperation: Exploiting Human Motor Skills with MARIOnET

Autonomous Agents & Multiagent Systems/Agent Theories, Architectures, and Languages, 2010

Annual Reviews in Control, 1996

Programming by Demonstration (PbD) is an intuitive method to program a robot. The user, acting as a teacher or programmer, shows how a particular task should be carried out. The demonstration is monitored using an interface device that ~d/ows the measurement and recording of both the applied commands as well as the data simultaneously perceived by robot's sensors. This paper identilies the kind of knowledge that the robot can actuul/y acquire from the human user through demonstrations and the requiroments that must be met/n order to be able to interpret what has been demonstrated. Finally, it presents and experimentally eva/uatss an approach to integrated acquisition, evaluation, tuning, and execution of elementary skills and ta~.levei programs for robots based on human demonstrations.

2007

Autonomous robots that can adapt to novel situations has been a long standing vision of robotics, artificial intelligence, and cognitive sciences. Early approaches to this goal during the heydays of artificial intelligence research in the late 1980s, however, made it clear that an approach purely based on reasoning or human insights would not be able to model all the perceptuomotor tasks that a robot should fulfill.

Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

This paper presents an approach for coupling robotic capability with human ability in dynamic motor skills, called "Human-Machine Mutual Actuation (HMMA)." We focus specifically on throwing motions and propose a method to control the release timing computationally. A system we developed achieves our concept, HMMA, by a robotic handheld device that acts as a release controller. We conducted user studies to validate the feasibility of the concept and clarify related technical issues to be tackled. We recognized that the system successfully performs on throwing according to the target while it exploits human ability. These empirical experiments suggest that robotic capability can be embedded into the users' motions without losing their senses of control. Throughout the user study, we also revealed several issues to be tackled in further research contributing to HMMA.

2021 IEEE International Conference on Robotics and Automation (ICRA), 2021

Imitation Learning (IL) is a powerful paradigm to teach robots to perform manipulation tasks by allowing them to learn from human demonstrations collected via teleoperation, but has mostly been limited to single-arm manipulation. However, many real-world tasks require multiple arms, such as lifting a heavy object or assembling a desk. Unfortunately, applying IL to multi-arm manipulation tasks has been challenging-asking a human to control more than one robotic arm can impose significant cognitive burden and is often only possible for a maximum of two robot arms. To address these challenges, we present MULTI-ARM ROBOTURK (MART), a multiuser data collection platform that allows multiple remote users to simultaneously teleoperate a set of robotic arms and collect demonstrations for multi-arm tasks. Using MART, we collected demonstrations for five novel two and three-arm tasks from several geographically separated users. From our data we arrived at a critical insight: most multi-arm tasks do not require global coordination throughout its full duration, but only during specific moments. We show that learning from such data consequently presents challenges for centralized agents that directly attempt to model all robot actions simultaneously, and perform a comprehensive study of different policy architectures with varying levels of centralization on our tasks. Finally, we propose and evaluate a base-residual policy framework that allows trained policies to better adapt to the mixed coordination setting common in multi-arm manipulation, and show that a centralized policy augmented with a decentralized residual model outperforms all other models on our set of benchmark tasks. Additional results and videos at https://roboturk. stanford.edu/multiarm

IEEE Spectrum, 2019

Sensors, 2022

Humans learn movements naturally, but it takes a lot of time and training to achieve expert performance in motor skills. In this review, we show how modern technologies can support people in learning new motor skills. First, we introduce important concepts in motor control, motor learning and motor skill learning. We also give an overview about the rapid expansion of machine learning algorithms and sensor technologies for human motion analysis. The integration between motor learning principles, machine learning algorithms and recent sensor technologies has the potential to develop AI-guided assistance systems for motor skill training. We give our perspective on this integration of different fields to transition from motor learning research in laboratory settings to real world environments and real world motor tasks and propose a stepwise approach to facilitate this transition.

Autonomous Robots, 2019

Teleoperating a robot for complex and intricate tasks demands a high mental workload from a human operator. Deploying multiple operators can mitigate this problem, but it can be also a costly solution. Learning from Demonstrations can reduce the human operator's burden by learning repetitive teleoperation tasks. Yet, the demonstrations via teleoperation tend to be inconsistent compared to other modalities of human demonstrations. In order to handle less consistent and asynchronous demonstrations effectively, this paper proposes a learning scheme based on Dynamic Movement Primitives. In particular, a new Expectation Maximization algorithm which synchronizes and encodes demonstrations with high temporal and spatial variances is proposed. Furthermore, we discuss two shared teleoperation architectures, where, instead of multiple human operators, a learned artificial agent and a human operator share authority over a task while teleoperating cooperatively. The agent controls the more mundane and repetitive motion in the task whereas human takes charge of the more critical and uncertain motion. The proposed algorithm together with the two shared teleoperation architectures (human-synchronized and agent-synchronized shared teleoperation) has been tested and validated through simulation and experiments on 3 Degrees-of-Freedom Phantom-to-Phantom teleoperation. Conclusively, the both proposed shared teleoperation architectures have shown superior performance when compared with the human-only teleoperation for a peg-in-hole task.

cetin.mericli.com

This paper describes a novel approach to robot teleoperation system capable of task learning through teleoperation. The system is based on the task rule extraction from the data acquired by monitoring a human operator during the teleoperation process. The basic information about teleoperation systems in general, applications and current trends are provided in the first part of the paper. The second part of this paper is dedicated to the architecture and software implementation of the proposed teleoperation system. The last sections of this paper are dedicated to the conducted experiments and conclusion of the project and the possible improvements considered for the future work.