Rodney Brooks (1991) put forth the idea that during an agent's interaction with its environment, ... more Rodney Brooks (1991) put forth the idea that during an agent's interaction with its environment, representations of the world often stand in the way. Instead, using the world as its own best model, i.e. interacting with it directly without making models, often leads to better and more natural behavior. The same perspective can be applied to representations of the agent's body. I analyze different examples from biology-octopus and humans in particular-and compare them with robots and their body models. At one end of the spectrum, the octopus, a highly intelligent animal, largely relies on the mechanical properties of its arms and peripheral nervous system. No central representations or maps of its body were found in its central nervous system. Primate brains do contain areas dedicated to processing body-related information and different body maps were found. Yet, these representations are still largely implicit and distributed and some functionality is also offloaded to the periphery. Robots, on the other hand, rely almost exclusively on their body models when planning and executing behaviors. I analyze the pros and cons of these different approaches and propose what may be the best solution for robots of the future.
Humans and animals excel in combining information from multiple sensory modalities, controlling t... more Humans and animals excel in combining information from multiple sensory modalities, controlling their complex bodies, adapting to growth, failures, or using tools. These capabilities are also highly desirable in robots. They are displayed by machines to some extent - yet, as is so often the case, the artificial creatures are lagging behind. The key foundation is an internal representation of the body that the agent - human, animal, or robot - has developed. In the biological realm, evidence has been accumulated by diverse disciplines giving rise to the concepts of body image, body schema, and others. In robotics, a model of the robot is an indispensable component that enables to control the machine. In this article I compare the character of body representations in biology with their robotic counterparts and relate that to the differences in performance that we observe. I put forth a number of axes regarding the nature of such body models: fixed vs. plastic, amodal vs. modal, explic...
Reaching to target locations on the body has been studied little despite its importance for adapt... more Reaching to target locations on the body has been studied little despite its importance for adaptive behaviors such as feeding, grooming, and indicating a source of discomfort. This behavior requires multisensory integration given that it involves coordination of touch, proprioception, and sometimes vision as well as action. Here we examined the origins of this skill by investigating how infants begin to localize targets on the body and the motor strategies by which they do so. Infants (7-21 months of age) were prompted to reach to a vibrating target placed at five arm/hand locations (elbow, crook of elbow, forearm, palm, and top of hand) one by one. To manually localize the target, infants needed to reach with one arm to the other. Results suggest that coordination increases with age in the strategies that infants used to localize body targets. Most infants showed bimanual coordination and usually moved the target arm toward the reaching arm to assist reaching. Furthermore, intersensory coordination increased with age. Simultaneous movements of the two arms increased with age, as did coordination between vision and reaching. The results provide new information about the development of multisensory integration during tactile localization and how such integration is linked to action.
End-effector extremes in operational space during avoidance and reaching
<p>For both the left and right end-effectors, the minimum and maximum values reached in the... more <p>For both the left and right end-effectors, the minimum and maximum values reached in the <i>x</i>−, <i>y</i>− and <i>z</i>− axis are shown, along with its range of operation. For safety reasons, the operational space of the robot was constrained within a sphere centered in the resting position (set to [−0.30, −0.20, +0.05] <i>m</i> for the left arm and [−0.30, +0.20, +0.05] <i>m</i> for the right arm—in iCub Root FoR) and with radius equal to 0.2 <i>m</i>. Please refer to <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0163713#pone.0163713.s001" target="_blank">S1 Fig</a> for a depiction of the robot’s kinematics during the avoidance and reaching scenario.</p
We studied the discrimination of deformable objects by grasping them using 4 different robot hand... more We studied the discrimination of deformable objects by grasping them using 4 different robot hands / grippers: Barrett hand (3 fingers with adjustable configuration, 96 tactile, 8 position, 3 torque sensors), qb SoftHand (5 fingers, 1 motor, position and current feedback), and two industrial type parallel jaw grippers with position and effort feedback (Robotiq 2F-85 and OnRobot RG6). A set of 9 ordinary objects differing in size and stiffness and another highly challenging set of 20 polyurethane foams differing in material properties only was used. We systematically compare the grippers' performance, together with the effects of: (1) type of classifier (k-NN, SVM, LSTM) operating on raw time series or on features, (2) action parameters (grasping configuration and speed of squeezing), (3) contribution of sensory modalities. Classification results are complemented by visualization of the data using PCA. We found: (i) all the grippers but the qb SoftHand could reliably distinguish the ordinary objects set; (ii) Barrett Hand reached around 95% accuracy on the foams; OnRobot RG6 around 75% and Robotiq 2F-85 around 70%; (iii) across all grippers, SVM over features and LSTM on raw time series performed best; (iv) faster compression speeds degrade classification performance; (v) transfer learning between compression speeds worked well for the Barrett Hand only; transfer between grasping configurations is limited; (vi) ablation experiments provided intriguing insights-sometimes a single sensory channel suffices for discrimination. Overall, the Barrett Hand as a complex and expensive device with rich sensory feedback provided best results, but uncalibrated parallel jaw grippers without tactile sensors can have sufficient performance for single-grasp object discrimination based on position and effort data only. Transfer learning between the different robot hands remains a challenge.
Soft electronic skins are one of the means to turn a classical industrial manipulator into a coll... more Soft electronic skins are one of the means to turn a classical industrial manipulator into a collaborative robot. For manipulators that are already fit for physical human-robot collaboration, soft skins can make them even safer. In this work, we study the after impact behavior of two collaborative manipulators (UR10e and KUKA LBR iiwa) and one classical industrial manipulator (KUKA Cybertech), in the presence or absence of an industrial protective skin (AIRSKIN). In addition, we isolate the effects of the passive padding and the active contribution of the sensor to robot reaction. We present a total of 2250 collision measurements and study the impact force, contact duration, clamping force, and impulse. This collected dataset is publicly available. We summarize our results as follows. For transient collisions, the passive skin properties lowered the impact forces by about 40 %. During quasistatic contact, the effect of skin covers-active or passive-cannot be isolated from the collision detection and reaction by the collaborative robots. Important effects of the stop categories triggered by the active protective skin were found. We systematically compare the different settings and compare the empirically established safe velocities with prescriptions by the ISO/TS 15066. In some cases, up to the quadruple of the ISO/TS 15066 prescribed velocity can comply with the impact force limits and thus be considered safe. We propose an extension of the formulas relating impact force and permissible velocity that take into account the stiffness and compressible thickness of the protective cover, leading to better predictions of the collision forces. At the same time, this work emphasizes the need for in situ measurements as all the factors we studied-presence of active/passive skin, safety stop settings, robot collision reaction, impact direction, and, of course, velocity-have effects on the force evolution after impact.
Humans and animals excel in combining information from multiple sensory modalities, controlling t... more Humans and animals excel in combining information from multiple sensory modalities, controlling their complex bodies, adapting to growth or failures, or using tools. The key foundation is an internal representation of the body that the agent—human, animal, or robot—has developed. In the biological realm, evidence has been accumulating in diverse disciplines, giving rise to the concepts of body image, body schema, and others. In robotics, a model of the robot is an indispensable component that enables to control the machine. This chapter compares the character of body representations in biology with their robotic counterparts and relates that to the differences in performance observed. Conclusions are drawn about how robots can inform the biological sciences dealing with body representations and which of the features of the ‘body in the brain’ should be transferred to robots, giving rise to more adaptive and resilient self-calibrating machines.
Recent advancements in object shape completion have enabled impressive object reconstructions usi... more Recent advancements in object shape completion have enabled impressive object reconstructions using only visual input. However, due to self-occlusion, the reconstructions have high uncertainty in the occluded object parts, which negatively impacts the performance of downstream robotic tasks such as grasping. In this work, we propose an active visuo-haptic shape completion method called Act-VH that actively computes where to touch the objects based on the reconstruction uncertainty. Act-VH reconstructs objects from point clouds and calculates the reconstruction uncertainty using IGR, a recent state-of-the-art implicit surface deep neural network. We experimentally evaluate the reconstruction accuracy of Act-VH against five baselines in simulation and in the real world. We also propose a new simulation environment for this purpose. The results show that Act-VH outperforms all baselines and that an uncertainty-driven haptic exploration policy leads to higher reconstruction accuracy than a random policy and a policy driven by Gaussian Process Implicit Surfaces. As a final experiment, we evaluate Act-VH and the best reconstruction baseline on grasping 10 novel objects. The results show that Act-VH reaches a significantly higher grasp success rate than the baseline on all objects. Together, this work opens up the door for using active visuo-haptic shape completion in more complex cluttered scenes.
2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), 2018
For robots to share the environment and cooperate with humans without barriers, we need to guaran... more For robots to share the environment and cooperate with humans without barriers, we need to guarantee safety to the operator and, simultaneously, to maximize the robot's usability. Safety is typically guaranteed by controlling the robot movements while, possibly, taking into account physical contacts with the operator, objects or tools. If possible, also the safety of the robot must be guaranteed. Not less importantly, as the complexity of the robots and their skills increase, usability becomes a concern. Social interaction technologies can save the day by enabling natural human-robot collaboration. In this paper we show a possible integration of physical and social Human-Robot Interaction methods (pHRI and sHRI respectively). Our reference task is object handover. We test both the case of the robot initiating the action and, vice versa, the robot receiving an object from the operator. Finally, we discuss possible extension with higher-level planning systems for added flexibility and reasoning skills.
2017 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), 2017
We have been observing how infants between 3 and 21 months react when a vibrotactile stimulation ... more We have been observing how infants between 3 and 21 months react when a vibrotactile stimulation (a buzzer) is applied to different parts of their bodies. Responses included in particular movement of the stimulated body part and successful reaching for and removal of the buzzer. Overall, there is a pronounced developmental progression from general to specific movement patterns, especially in the first year. In this article we review the series of studies we conducted and then focus on possible mechanisms that might explain what we observed. One possible mechanism might rely on the brain extracting "sensorimotor contingencies" linking motor actions and resulting sensory consequences. This account posits that infants are driven by intrinsic motivation that guides exploratory motor activity, at first generating random motor babbling with self-touch occurring spontaneously. Later goal-oriented motor behavior occurs, with self-touch as a possible effective tool to induce informative contingencies. We connect this sensorimotor view with a second possible account that appeals to the neuroscientific concepts of cortical maps and coordinate transformations. In this second account, the improvement of reaching precision is mediated by refinement of neuronal maps in primary sensory and motor cortices-the homunculi-as well as in frontal and parietal cortical regions dedicated to sensorimotor processing. We complement this theoretical account with modeling on a humanoid robot with artificial skin where we implemented reaching for tactile stimuli as well as learning the "somatosensory homunculi". We suggest that this account can be extended to reflect the driving role of sensorimotor contingencies in human development. In our conclusion we consider possible extensions of our current experiments which take account of predictions derived from both these kinds of models.
2021 IEEE International Conference on Robotics and Automation (ICRA), 2021
Collaborative robots, i.e. robots designed for direct interaction with a human, present a promisi... more Collaborative robots, i.e. robots designed for direct interaction with a human, present a promising step in robotic manufacturing. However, their performance is limited by the safety demands of standards. In this article, we measure the forces exerted by two robot arms (UR10e and Kuka LBR iiwa) on an impact measuring device in different positions in the robot workspace and with various velocities. Based on these measurements, we investigate the Power and Force Limiting regime presented in ISO/TS 15066. Impact forces are in practice hard to calculate analytically as many properties of the robots are not available (e.g., proprietary control algorithms). This motivates the use of simple, yet reasonably accurate, approximations. Our results show that height of the impact location is also an important factor and that an accurate model of the robot can be created from a limited number of impact samples. Previous work predicted impact forces based on other factors (distance, velocity, weight), yet these predictions are less accurate. This would allow a fast estimation of the impact forces in the robot's workspace and thus make it easier to design a safe human-robot collaboration setup.
Artificial Neural Networks and Machine Learning – ICANN 2017, 2017
The space immediately surrounding our body, or peripersonal space, is crucial for interaction wit... more The space immediately surrounding our body, or peripersonal space, is crucial for interaction with the environment. In primate brains, specific neural circuitry is responsible for its encoding. An important component is a safety margin around the body that draws on visuo-tactile interactions: approaching stimuli are registered by vision and processed, producing anticipation or prediction of contact in the tactile modality. The mechanisms of this representation and its development are not understood. We propose a computational model that addresses this: a neural network composed of a Restricted Boltzmann Machine and a feedforward neural network. The former learns in an unsupervised manner to represent position and velocity features of the stimulus. The latter is trained in a supervised way to predict the position of touch (contact). Unique to this model, it considers: (i) stimulus position and velocity, (ii) uncertainty of all variables, and (iii) not only multisensory integration but also prediction.
IEEE Transactions on Cognitive and Developmental Systems, 2021
An early integration of tactile sensing into motor coordination is the norm in animals, but still... more An early integration of tactile sensing into motor coordination is the norm in animals, but still a challenge for robots. Tactile exploration through touches on the body gives rise to first body models and bootstraps further development such as reaching competence. Reaching to one's own body requires connections of the tactile and motor space only. Still, the problems of high dimensionality and motor redundancy persist. Through an embodied computational model for the learning of self-touch on a simulated humanoid robot with artificial sensitive skin, we demonstrate that this task can be achieved (i) effectively and (ii) efficiently at scale by employing the computational frameworks for the learning of internal models for reaching: intrinsic motivation and goal babbling. We relate our results to infant studies on spontaneous body exploration as well as reaching to vibrotactile targets on the body. We analyze the reaching configurations of one infant followed weekly between 4 and 18 months of age and derive further requirements for the computational model: accounting for (iii) continuous rather than sporadic touch and (iv) consistent redundancy resolution. Results show the general success of the learning models in the touch domain, but also point out limitations in achieving fully continuous touch.
2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019
So-called collaborative robots are a current trend in industrial robotics. However, they still fa... more So-called collaborative robots are a current trend in industrial robotics. However, they still face many problems in practical application such as reduced speed to ascertain their collaborativeness. The standards prescribe two regimes: (i) speed and separation monitoring and (ii) power and force limiting, where the former requires reliable estimation of distances between the robot and human body parts and the latter imposes constraints on the energy absorbed during collisions prior to robot stopping. Following the standards, we deploy the two collaborative regimes in a single application and study the performance in a mock collaborative task under the individual regimes, including transitions between them. Additionally, we compare the performance under "safety zone monitoring" with keypoint pair-wise separation distance assessment relying on an RGB-D sensor and skeleton extraction algorithm to track human body parts in the workspace. Best performance has been achieved in the following setting: robot operates at full speed until a distance threshold between any robot and human body part is crossed; then, reduced robot speed per power and force limiting is triggered. Robot is halted only when the operator's head crosses a predefined distance from selected robot parts. We demonstrate our methodology on a setup combining a KUKA LBR iiwa robot, Intel RealSense RGB-D sensor and OpenPose for human pose estimation.
Representation and Reality in Humans, Other Living Organisms and Intelligent Machines, 2017
Engineers fine-tune the design of robot bodies for control purposes, however, a methodology or se... more Engineers fine-tune the design of robot bodies for control purposes, however, a methodology or set of tools is largely absent, and optimization of morphology (shape, material properties of robot bodies, etc.) is lagging behind the development of controllers. This has become even more prominent with the advent of compliant, deformable or "soft" bodies. These carry substantial potential regarding their exploitation for control-sometimes referred to as "morphological computation". In this article 1 , we briefly review different notions of computation by physical systems and propose the dynamical systems framework as the most useful in the context of describing and eventually designing the interactions of controllers and bodies. Then, we look at the pros and cons of simple vs. complex bodies, critically reviewing the attractive notion of "soft" bodies automatically taking over control tasks. We address another key dimension of the design space-whether model-based control should be used and to what extent it is feasible to develop faithful models for different morphologies.
In primate brains, tactile and proprioceptive inputs are relayed to the somatosensory cortex whic... more In primate brains, tactile and proprioceptive inputs are relayed to the somatosensory cortex which is known for somatotopic representations, or, "homunculi". Our research centers on understanding the mechanisms of the formation of these and more higher-level body representations (body schema) by using humanoid robots and neural networks to construct models. We specifically focus on how spatial representation of the body may be learned from somatosensory information in self-touch configurations. In this work, we target the representation of proprioceptive inputs, which we take to be joint angles in the robot. The inputs collected in different body postures serve as inputs to a Self-Organizing Map (SOM) with a 2D lattice on the output. With unrestricted, all-to-all connections, the map is not capable of representing the input space while preserving the topological relationships, because the intrinsic dimensionality of the body posture space is too large. Hence, we use a meth...
2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), 2019
The development of reaching in infants has been studied for nearly nine decades. Originally, it w... more The development of reaching in infants has been studied for nearly nine decades. Originally, it was thought that early reaching is visually guided, but more recent evidence is suggestive of "visually elicited" reaching, i.e. infant is gazing at the object rather than its hand during the reaching movement. The importance of haptic feedback has also been emphasized. Inspired by these findings, in this work we use the simulated iCub humanoid robot to construct a model of reaching development. The robot is presented with different objects, gazes at them, and performs motor babbling with one of its arms. Successful contacts with the object are detected through tactile sensors on hand and forearm. Such events serve as the training set, constituted by images from the robot's two eyes, head joints, tactile activation, and arm joints. A deep neural network is trained with images and head joints as inputs and arm configuration and touch as output. After learning, the network can successfully infer arm configurations that would result in a successful reach, together with prediction of tactile activation (i.e. which body part would make contact). Our main contribution is twofold: (i) our pipeline is endto-end from stereo images and head joints (6 DoF) to armtorso configurations (10 DoF) and tactile activations, without any preprocessing, explicit coordinate transformations etc.; (ii) unique to this approach, reaches with multiple effectors corresponding to different regions of the sensitive skin are possible.
Somatosensory inputs can be grossly divided into tactile (or cutaneous) and proprioceptive -- the... more Somatosensory inputs can be grossly divided into tactile (or cutaneous) and proprioceptive -- the former conveying information about skin stimulation, the latter about limb position and movement. The principal proprioceptors are constituted by muscle spindles, which deliver information about muscle length and speed. In primates, this information is relayed to the primary somatosensory cortex and eventually the posterior parietal cortex, where integrated information about body posture (postural schema) is presumably available. However, coming from robotics and seeking a biologically motivated model that could be used in a humanoid robot, we faced a number of difficulties. First, it is not clear what neurons in the ascending pathway and primary somatosensory cortex code. To an engineer, joint angles would seem the most useful variables. However, the lengths of individual muscles have nonlinear relationships with the angles at joints. Kim et al. (Neuron, 2015) found different types of ...
2020 Joint IEEE 10th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), 2020
The mechanisms of infant development are far from understood. Learning about one's own body is li... more The mechanisms of infant development are far from understood. Learning about one's own body is likely a foundation for subsequent development. Here we look specifically at the problem of how spontaneous touches to the body in early infancy may give rise to first body models and bootstrap further development such as reaching competence. Unlike visually elicited reaching, reaching to own body requires connections of the tactile and motor space only, bypassing vision. Still, the problems of high dimensionality and redundancy of the motor system persist. In this work, we present an embodied computational model on a simulated humanoid robot with artificial sensitive skin on large areas of its body. The robot should autonomously develop the capacity to reach for every tactile sensor on its body. To do this efficiently, we employ the computational framework of intrinsic motivations and variants of goal babbling-as opposed to motor babbling-that prove to make the exploration process faster and alleviate the ill-posedness of learning inverse kinematics. Based on our results, we discuss the next steps in relation to infant studies: what information will be necessary to further ground this computational model in behavioral data.
Uploads
Papers by Matej Hoffmann