Skip to main content

Reza Azadeh

University of Massachusetts, Lowell, Computer Science, Faculty Member

Università degli Studi di Genova, Dibris, PhD Student in Robotics

Georgia Institute of Technology, College of Computing, Post-Doc

Followers

37

Following

21

Co-authors

7

Public Views

I am an Assistant Professor with the Department of Computer Science at the University of Massachusetts Lowell. I received my PhD in "Robotics, Cognition, and Interaction Technologies" from the University of Genoa in collaboration with Italian Institute of Technology. My research interests include Robot Learning, Reinforcement Learning, Visuospatial Skill Learning, Imitation Learning, and Reactive Behavior Learning.
For more information please visit my personal website: http://www.ahmadzadeh.info/
Address: 801 Atlantic DR NW,
College of Computing,
30332, Atlanta, GA

less

InterestsView All (16)

Uploads

Papers by Reza Azadeh

Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance

We contribute a study benchmarking the performance of multiple motion-based learning from demonst... more We contribute a study benchmarking the performance of multiple motion-based learning from demonstration approaches. Given the number and diversity of existing methods, it is critical that comprehensive empirical studies be performed comparing the relative strengths of these techniques. In particular, we evaluate four approaches based on properties an end user may desire for real-world tasks. To perform this evaluation, we collected data from nine participants, across four manipulation tasks. The resulting demonstrations were used to train 180 task models and evaluated on 720 task reproductions on a physical robot. Our results detail how i) complexity of the task, ii) the expertise of the human demonstrator, and iii) the starting configuration of the robot affect task performance. The collected dataset of demonstrations, robot executions, and evaluations are publicly available. Research insights and guidelines are also provided to guide future research and deployment choices about these approaches.

Reward-Sharing Relational Networks in Multi-Agent Reinforcement Learning as a Framework for Emergent Behavior

by Reza Azadeh and Kshitij Jerath

Adaptive and Learning Agents (ALA) at AAMAS, 2021

In this work, we integrate 'social' interactions into the MARL setup through a user-defined relat... more In this work, we integrate 'social' interactions into the MARL setup through a user-defined relational network and examine the effects of agent-agent relations on the rise of emergent behaviors. Leveraging insights from sociology and neuroscience, our proposed framework models agent relationships using the notion of Reward-Sharing Relational Networks (RSRN), where network edge weights act as a measure of how much one agent is invested in the success of (or 'cares about') another. We construct relational rewards as a function of the RSRN interaction weights to collectively train the multi-agent system via a multi-agent reinforcement learning algorithm. The performance of the system is tested for a 3-agent scenario with different relational network structures (e.g., self-interested, communitarian, and authoritarian networks). Our results indicate that reward-sharing relational networks can significantly influence learned behaviors. We posit that RSRN can act as a framework where different relational networks produce distinct emergent behaviors, often analogous to the intuited sociological understanding of such networks.

Towards Robust Skill Generalization: Unifying Learning from Demonstration and Motion Planning

In this paper, we present Combined Learning from demonstration And Motion Planning (CLAMP) as an ... more In this paper, we present Combined Learning from demonstration And Motion Planning (CLAMP) as an efficient approach to skill learning and gener-alizable skill reproduction. CLAMP combines the strengths of Learning from Demonstration (LfD) and motion planning into a unifying framework. We carry out probabilistic inference to find trajectories which are optimal with respect to a given skill and also feasible in different scenarios. We use factor graph optimization to speed up inference. To encode optimality, we provide a new probabilistic skill model based on a stochastic dynamical system. This skill model requires minimal parameter tuning to learn, is suitable to encode skill constraints, and allows efficient inference. Preliminary experimental results showing skill generalization over initial robot state and unforeseen obstacles are presented.

Visuospatial Skill Learning for Robots

A novel skill learning approach is proposed that allows a robot to acquire human-like visuospatia... more A novel skill learning approach is proposed that allows a robot to acquire human-like visuospatial skills for object manipulation tasks. Visuospatial skills are attained by observing spatial relationships among objects through demonstrations. The proposed Visuospatial Skill Learning (VSL) is a goal-based approach that focuses on achieving a desired goal configuration of objects relative to one another while maintaining the sequence of operations. VSL is capable of learning and generalizing multi-operation skills from a single demonstration, while requiring minimum prior knowledge about the objects and the environment. In contrast to many existing approaches, VSL offers simplicity, efficiency and user-friendly human-robot interaction. We also show that VSL can be easily extended towards 3D object manipulation tasks, simply by employing point cloud processing techniques. In addition, a robot learning framework, VSL-SP, is proposed by integrating VSL, Imitation Learning, and a conventional planning method. In VSL-SP, the sequence of performed actions are learned using VSL, while the sensorimotor skills are learned using a conventional trajectory-based learning approach. such integration easily extends robot capabilities to novel situations, even by users without programming ability. In VSL-SP the internal planner of VSL is integrated with an existing action-level symbolic planner. Using the underlying constraints of the task and extracted symbolic predicates, identified by VSL, symbolic representation of the task is updated. Therefore the planner maintains a generalized representation of each skill as a reusable action, which can be used in planning and performed independently during the learning phase. The proposed approach is validated through several real-world experiments.

Social Gaze Behavior for Face to Face Human-Robot Interaction

This short paper discusses the importance of human-like gaze behaviors for humanoid robots with p... more This short paper discusses the importance of human-like gaze behaviors for humanoid robots with physical eyes. It gives a brief overview of the functions gaze fulfills in human-human interactions from a social evolutionary perspective. In the second part we describe how human-like gaze has been implemented in robot in the research field of social robotics in the last years. The last part of the paper briefly introduces an architecture for a conversational gaze controller (CGC). The parameters of this gaze controller are based on the analysis of a large corpus of gaze tracking data collected during human-human conversations. We describe our experimental approach for obtaining this data and discuss the implications of endowing robots with human-like behaviors and enabling them to engage in face to face social interactions.

Trajectory Learning from Demonstration with Canal Surfaces: A Parameter-free Approach

by Reza Azadeh and Roshni Kaushik

We present a novel geometric framework for intuitively encoding and learning a wide range of traj... more We present a novel geometric framework for intuitively encoding and learning a wide range of trajectory-based skills from human demonstrations. Our approach identifies and extracts the main characteristics of the demonstrated skill, which are spatial correlations across different demonstrations. Using the extracted characteristics, the proposed approach generates a continuous representation of the skill based on the concept of canal surfaces. Canal surfaces are Euclidean surfaces formed as the envelope of a family of regular surfaces (e.g. spheres) whose centers lie on a space curve. The learned skill can be reproduced, as a time-independent trajectory, and generalized to unforeseen situations inside the canal while its main characteristics are preserved. The main advantages of the proposed approach include: (a) requiring no parameter tuning, (b) maintaining the main characteristics and implicit boundaries of the skill, and (c) generalizing the learned skill over the initial condition of the movement, while exploiting the whole demonstration space to reproduce a variety of successful movements. Evaluations using simulated and real-world data exemplify the feasibility and robustness of our approach.

Learning reactive robot behavior for autonomous valve turning

by Rodrigo Jamisola and Reza Azadeh

2014 IEEE-RAS International Conference on Humanoid Robots, 2014

A learning approach is proposed for the challenging task of autonomous robotic valve turning in t... more A learning approach is proposed for the challenging task of autonomous robotic valve turning in the presence of active disturbances and uncertainties. The valve turning task comprises two phases: reaching and turning. For the reaching phase the manipulator learns how to generate trajectories to reach or retract from the target. The learning is based on a set of trajectories demonstrated in advance by the operator. The turning phase is accomplished using a hybrid force/motion control strategy. Furthermore, a reactive decision making system is devised to react to the disturbances and uncertainties arising during the valve turning process. The reactive controller monitors the changes in force, movement of the arm with respect to the valve, and changes in the distance to the target. Observing the uncertainties, the reactive system modulates the valve turning task by changing the direction and rate of the movement. A real-world experiment with a robot manipulator mounted on a movable base is conducted to show the efficiency and validity of the proposed approach.

On-line Learning to Recover from Thruster Failures on Autonomous Underwater Vehicles

by Reza Azadeh and Petar Kormushev

2013 Oceans San Diego, 2013

We propose a method for computing on-line the controller of an Autonomous Underwater Vehicle unde... more We propose a method for computing on-line the controller of an Autonomous Underwater Vehicle under thruster failures. The method is general and can be applied to both redundant and under-actuated AUVs, as it does not rely on the modification of the thruster control matrix. We define an optimization problem on a specific class of functions, in order to compute the optimal control law that achieves the target without using the faulty thruster. The method is framed within modelbased policy search for reinforcement learning, and we study its applicability on the model of the AUV Girona500. We performed experiments with policies of increasing complexity, testing the on-line feasibility of the approach as the optimization problem becomes more complex.

Online Direct Policy Search for Thruster Failure Recovery in Autonomous Underwater Vehicles

Autonomous underwater vehicles are prone to various factors that may lead a mission to fail and c... more Autonomous underwater vehicles are prone to various factors that may lead a mission to fail and cause unrecoverable damages. Even robust controllers cannot make sure that the robot is able to navigate to a safe location in such situations. In this paper we propose an online learning method for reconfiguring the controller, which tries to recover the robot and survive the mission using the current asset of the system. The proposed method is framed in the reinforcement learning setting, and in particular as a model-based direct policy search approach. Since learning on a damaged vehicle would be impossible owing to time and energy constraints, learning is performed on a model which is identified and kept updated online. We evaluate the applicability of our method with different policy representations and learning algorithms, on the model of the vehicle Girona500.

On-line Learning to Recover from Thruster Failures on Autonomous Underwater Vehicles

We propose a method for computing on-line the controller of an Autonomous Underwater Vehicle unde... more We propose a method for computing on-line the controller of an Autonomous Underwater Vehicle under thruster failures. The method is general and can be applied to both redundant and under-actuated AUVs, as it does not rely on the modification of the thruster control matrix. We define an optimization problem on a specific class of functions, in order to compute the optimal control law that achieves the target without using the faulty thruster. The method is framed within modelbased policy search for reinforcement learning, and we study its applicability on the model of the AUV Girona500. We performed experiments with policies of increasing complexity, testing the on-line feasibility of the approach as the optimization problem becomes more complex.

Multi-Objective Reinforcement Learning for AUV Thruster Failure Recovery

by Reza Azadeh and Petar Kormushev

This paper investigates learning approaches for discovering fault-tolerant control policies to ov... more This paper investigates learning approaches for discovering fault-tolerant control policies to overcome thruster failures in Autonomous Underwater Vehicles (AUV). The proposed approach is a model-based direct policy search that learns on an on-board simulated model of the vehicle. When a fault is detected and isolated the model of the AUV is reconfigured according to the new condition. To discover a set of optimal solutions a multi-objective reinforcement learning approach is employed which can deal with multiple conflicting objectives. Each optimal solution can be used to generate a trajectory that is able to navigate the AUV towards a specified target while satisfying multiple objectives. The discovered policies are executed on the robot in a closed-loop using AUV's state feedback. Unlike most existing methods which disregard the faulty thruster, our approach can also deal with partially broken thrusters to increase the persistent autonomy of the AUV. In addition, the proposed approach is applicable when the AUV either becomes under-actuated or remains redundant in the presence of a fault. We validate the proposed approach on the model of the Girona500 AUV.

Modal Approach to Modeling of Hyper-Redundant Robot Manipulators Dynamics and Design of Fuzzy Controller for the System

Hyper-redundant manipulators having degrees of freedom much more than required, have many advanta... more Hyper-redundant manipulators having degrees of freedom much more than required, have many advantages and
important capabilities. Efficient and proper industrial usage of these robots is depended on their controller ability and a
deep knowledge of the system dynamics is required to design the controller. In this thesis, modeling of the manipulator
dynamics is done using a special curve called ‘backbone curve’, modal method, Lagrangian mechanics and geometric
transformation between variables spaces of backbone curve and manipulator joints. To identify the backbone curve
that captures all global geometric characteristics of the hyper-redundant manipulator features, harmonic modes are
used for curvature function. The dynamics equations of the hyper-redundant manipulator are derived in the joint
variables space by Lagrangian mechanics and transformed to the backbone curve variables space. The dependency
of the Nonlinear and coupled terms of the dynamics model to joint variables makes some difficulties in classical
methods to controller design. To overcome this problem, fuzzy controllers that have appropriate efficiency in complex
and nonlinear systems are used. To evaluate the fuzzy control method based on dynamics model, some traditional and
fuzzy controllers are designed for single and two degrees of freedom manipulators and their efficiency investigated.
The results show the desired condition of the fuzzy controller compared to the other controllers. For hyper-redundant
manipulators which are high order multivariable nonlinear systems and have coupled states, the fuzzy controllers
indicate some advantages when compared to the other classical controllers. For demonstrating this matter, dynamics
modeling of 10 degrees of freedom manipulator is done. Then a fuzzy controller is designed with attention to the
dynamics behavior of the system. Manipulator behavior through various and noisy inputs are evaluated by simulation
of the model including fuzzy controller. The results show very small error in manipulator motions and suitable
condition of the designed fuzzy controller based on the dynamics model.

Learning Symbolic Representations of Actions from Human Demonstrations

by Reza Azadeh, Fulvio Mastrogiovanni, and Petar Kormushev

Abstract— In this paper, a robot learning approach is proposed which integrates Visuospatial Skil... more Abstract— In this paper, a robot learning approach is proposed which integrates Visuospatial Skill Learning, Imitation Learning, and conventional planning methods. In our approach, the sensorimotor skills (i.e., actions) are learned through a learning from demonstration strategy. The sequence of performed actions is learned through demonstrations using Visuospatial Skill Learning. A standard action-level planner is used to represent a symbolic description of the skill, which allows the system to represent the skill in a discrete, symbolic form. The Visuospatial Skill Learning module identifies the underlying constraints of the task and extracts symbolic predicates (i.e., action preconditions and effects), thereby updating the planner representation while the skills are being learned. Therefore the planner maintains a generalized representation of each skill as a reusable action, which can be planned and performed independently during the learning phase. Preliminary experimental results on the iCub robot are presented.

Learning reactive robot behavior for autonomous valve turning

by Rodrigo Jamisola, Reza Azadeh, and Petar Kormushev

A learning approach is proposed for the challenging task of autonomous robotic valve turning in t... more A learning approach is proposed for the challenging task of autonomous robotic valve turning in the presence of active disturbances and uncertainties. The valve turning task comprises two phases: reaching and turning. For the reaching phase the manipulator learns how to generate trajectories to reach or retract from the target. The learning is based on a set of trajectories demonstrated in advance by the operator. The turning phase is accomplished using a hybrid force/motion control strategy. Furthermore, a reactive decision making system is devised to react to the disturbances and uncertainties arising during the valve turning process. The reactive controller monitors the changes in force, movement of the arm with respect to the valve, and changes in the distance to the target. Observing the uncertainties, the reactive system modulates the valve turning task by changing the direction and rate of the movement. A real-world experiment with a robot manipulator mounted on a movable base is conducted to show the efficiency and validity of the proposed approach.

Autonomous Robotic Valve Turning: A Hierarchical Learning Approach

Autonomous valve turning is an extremely challenging task for an Autonomous Underwater Vehicle (... more Autonomous valve turning is an extremely challenging task for an Autonomous Underwater Vehicle (AUV). To resolve this challenge, this paper proposes a set of different computational techniques integrated in a three-layer hierarchical scheme. Each layer realizes speciﬁc subtasks to improve the persistent autonomy of the system. In the ﬁrst layer, the robot acquires the motor skills of approaching and grasping the valve by kinesthetic teaching. A Reactive Fuzzy Decision Maker (RFDM) is devised in the second layer which reacts to the relative movement between the valve and the AUV, and alters the robot’s movement accordingly. Apprenticeship learning method, implemented in the third layer, performs tuning of the RFDM based on expert knowledge. Although the long-term goal is to perform the valve turning task on a real AUV, as a ﬁrst step the proposed approach is tested in a laboratory environment.

Interactive Robot Learning of Visuospatial Skills

This paper proposes a novel interactive robot learning approach for acquiring visuospatial skill... more This paper proposes a novel interactive robot learning approach for acquiring visuospatial skills. It allows a robot to acquire new capabilities by observing a demonstration while interacting with a human caregiver. Most existing learning from demonstration approaches focus on the trajectories, whereas in our approach the focus is placed on achieving a desired goal configuration of objects relative to one another. Our approach is based on visual perception which captures the object’s context for each demonstrated action. The context embodies implicitly the visuospatial representation including the relative positioning of the object with respect to multiple other objects simultaneously. The proposed approach is capable of learning and generalizing different skills such as object reconfiguration, classification, and turn-taking interaction. The robot learns to achieve the goal from a single demonstration while requiring minimum a priori knowledge about the environment. We illustrate the capabilities of our approach using four real world experiments with a Barrett WAM robot.

Covariance Analysis as a Measure of Policy Robustness in Reinforcement Learning

In this paper we propose covariance analysis as a metric for reinforcement learning to improve th... more In this paper we propose covariance analysis as a metric for reinforcement learning to improve the robustness of a learned policy. The local optima found during the exploration are analyzed in terms of the total cumulative reward and the local behavior of the system in the neighborhood of the optima. The analysis is performed in the solution space to select a policy that exhibits robustness in uncertain and noisy environments. We demonstrate the utility of the method using our previously developed system where an autonomous underwater vehicle (AUV) has to recover from a thruster failure. When a failure is detected the recovery system is invoked, which uses simulations to learn a new controller that utilizes the remaining functioning thrusters to achieve the goal of the AUV, that is, to reach a target position. In this paper, we use covariance analysis to examine the performance of the top, n, policies output by the previous algorithm. We propose a scoring metric that uses the output of the covariance analysis, the time it takes the AUV to reach the target position and the distance between the target position and the AUV's final position. The top polices are simulated in a noisy environment and evaluated using the proposed scoring metric to analyze the effect of noise on their performance. The policy that exhibits more tolerance to noise is selected. We show experimental results where covariance analysis successfully selects a more robust policy that was ranked lower by the original algorithm.

Towards Autonomous Robotic Valve Turning

by A. Carrera, Reza Azadeh, and D. Caldwell

In this paper an autonomous intervention robotic task to learn the skill of grasping and turning ... more In this paper an autonomous intervention robotic task to learn the skill of grasping and turning a valve is described. To resolve this challenge a set of different techniques are proposed, each one realizing a specific task and sending information to the others in a Hardware-In-Loop (HIL) simulation. To improve the estimation of the valve position, an Extended Kalman Filter is designed. Also to learn the trajectory to follow with the robotic arm, Imitation Learning approach is used. In addition, to perform safely the task a fuzzy system is developed which generates appropriate decisions. Although the achievement of this task will be used in an Autonomous Underwater Vehicle, for the first step this idea has been tested in a laboratory environment with an available robot and a sensor.

Modeling of hyper-redundant manipulators dynamics and design of fuzzy controller for the system

Hyper-redundant manipulators having degrees of freedom much more than required, have many advanta... more Hyper-redundant manipulators having degrees of freedom much more than required, have many advantages and important capabilities. In this paper, modeling of the manipulator dynamics is done using a special curve called 'backbone' curve, modal method, Lagrangian mechanics and geometric transformation between variables space of backbone curve and manipulator joints. The dependency of the nonlinear and coupled terms of the dynamics model to joint variables makes some difficulties in classical methods to controller design. To overcome this problem, fuzzy controllers that have appropriate efficiency in complex and nonlinear systems are used. For demonstrating this matter, dynamics modeling of 10 degrees of freedom manipulator is done. Then a fuzzy controller is designed with attention to the dynamics behavior of the system. Manipulator behavior through various and noisy inputs are evaluated by simulation of the model including fuzzy controller. The results show very small error in manipulator motions and suitable condition of the designed fuzzy controller based on the dynamics model. a b c KIMAS 2005 WALTHAM, MA, USA KIMAS 2005, April 18-21, 2005, Waltham, MA, USA. 0-7803-9013-X/05/$20.00

Online Discovery of AUV Control Policies to Overcome Thruster Failures

We investigate methods to improve fault-tolerance of Autonomous Underwater Vehicles (AUVs) to inc... more We investigate methods to improve fault-tolerance of Autonomous Underwater Vehicles (AUVs) to increase their reliability and persistent autonomy. We propose a learning-based approach that is able to discover new control policies to overcome thruster failures as they happen. The proposed approach is a model-based direct policy search that learns on an on-board simulated model of the AUV. The model is adapted to a new condition when a fault is detected and isolated. Since the approach generates an optimal trajectory, the learned fault-tolerant policy is able to navigate the AUV towards a specified target with minimum cost. Finally, the learned policy is executed on the real robot in a closed-loop using the state feedback of the AUV. Unlike most existing methods which rely on the redundancy of thrusters, our approach is also applicable when the AUV becomes under-actuated in the presence of a fault. To validate the feasibility and efficiency of the presented approach, we evaluate it with three learning algorithms and three policy representations with increasing complexity. The proposed method is tested on a real AUV, Girona500.