Mehdi Khamassi

Sorbonne University, Institut des Systèmes Intelligents et de Robotique, Faculty Member

Centre National de la Recherche Scientifique, Institut des Systèmes Intelligents et de Robotique, Faculty Member

Followers

213

Following

106

Co-authors

Public Views

Dr. Mehdi Khamassi is currently a research director at the French National Center for Scientific Research (CNRS) in the Institute of Intelligent Systems and Robotics (ISIR) at Sorbonne Universités / Université Pierre et Marie Curie - Paris 6 (UPMC). He has been a permanent research scientist at the CNRS since 2010. In the meantime, he has also been a visiting academic at the Center for Mind/Brain Sciences of the University of Trento, Italy, at the Robotics Lab of the National Technical University of Athens, Greece, and at the Department of Experimental Psychology of the University of Oxford, UK. He obtained his Habilitation to Direct Researches in Biology from UPMC in 2014. His research interests include decision-making, reinforcement learning, performance monitoring, interaction and reward signals in social and non-social contexts.
Address: ISIR - Université Pierre et Marie Curie
Pyramide, Tour 55 - Boîte courrier 173
4 place Jussieu, 75252 Paris Cedex 05, France

less

Interests

Uploads

Papers by Mehdi Khamassi

Modelling individual differences in rats using a dual learning systems approach and factored representations

Coordination of parallel learning processes in animals and robots

Download

Complementary roles of the rat prefrontal cortex and striatum in reward-based learning and shifting navigation strategies: Electrophysiological and computational studies, application to simulated autonomous robotics

Complementary roles of the rat prefrontal cortex and striatum in rewardbased learning and shiftin... more

Download

Theta Band LFP Coherence Between Hippocampus And Prefrontal Cortex and Reorganization of Ensemble Cell Activity During Learning

Time course of reactivation of memory-related cell ensembles in the rat medial prefrontal cortex during sleep

Design of a Control Architecture for Habit Learning in Robots

Lecture Notes in Computer Science, 2014

Researches in psychology and neuroscience have identified multiple decision systems in mammals, e... more Researches in psychology and neuroscience have identified multiple decision systems in mammals, enabling control of behavior to shift with training and familiarity of the environment from a goal-directed system to a habitual system. The former relies on the explicit estimation of future consequences of actions through planning towards a particular goal, which makes decision time longer but produces rapid adaptation to changes in the environment. The latter learns to associate values to particular stimulus-response associations, leading to quick reactive decisionmaking but slow relearning in response to environmental changes. Computational neuroscience models have formalized this as a coordination of model-based and model-free reinforcement learning. From this inspiration we hypothesize that it could enable robots to learn habits, detect when these habits are appropriate and thus avoid long and costly computations of the planning system. We illustrate this in a simple repetitive cube-pushing task on a conveyor belt, where a speed-accuracy trade-off is required. We show that the two systems have complementary advantages in these tasks, which can be combined for performance improvement.

Download

Experimental predictions drawn from a computational model of sign-trackers and goal-trackers

Journal of Physiology-Paris, 2014

Article history: Available online xxxx Keywords: Reinforcement learning Dopamine Pavlovian condit... more

Download

Accounting for Negative Automaintenance in Pigeons: A Dual Learning Systems Approach and Factored Representations

PLoS ONE, 2014

Animals, including Humans, are prone to develop persistent maladaptive and suboptimal behaviours.... more Animals, including Humans, are prone to develop persistent maladaptive and suboptimal behaviours. Some of these behaviours have been suggested to arise from interactions between brain systems of Pavlovian conditioning, the acquisition of responses to initially neutral stimuli previously paired with rewards, and instrumental conditioning, the acquisition of active behaviours leading to rewards. However the mechanics of these systems and their interactions are still unclear. While extensively studied independently, few models have been developed to account for these interactions. On some experiment, pigeons have been observed to display a maladaptive behaviour that some suggest to involve conflicts between Pavlovian and instrumental conditioning. In a procedure referred as negative automaintenance, a key light is paired with the subsequent delivery of food, however any peck towards the key light results in the omission of the reward. Studies showed that in such procedure some pigeons persisted in pecking to a substantial level despite its negative consequence, while others learned to refrain from pecking and maximized their cumulative rewards. Furthermore, the pigeons that were unable to refrain from pecking could nevertheless shift their pecks towards a harmless alternative key light. We confronted a computational model that combines dual-learning systems and factored representations, recently developed to account for sign-tracking and goal-tracking behaviours in rats, to these negative automaintenance experimental data. We show that it can explain the variability of the observed behaviours and the capacity of alternative key lights to distract pigeons from their detrimental behaviours. These results confirm the proposed model as an interesting tool to reproduce experiments that could involve interactions between Pavlovian and instrumental conditioning. The model allows us to draw predictions that may be experimentally verified, which could help further investigate the neural mechanisms underlying theses interactions.

Download

Coordination of adaptive working memory and reinforcement learning systems explaining choice and reaction time in a human experiment

BMC Neuroscience, 2014

Medial prefrontal cortex and the adaptive regulation of reinforcement learning parameters

Converging evidence suggest that the medial prefrontal cortex (MPFC) is involved in feedback cate... more Converging evidence suggest that the medial prefrontal cortex (MPFC) is involved in feedback categorization, performance monitoring, and task monitoring, and may contribute to the online regulation of reinforcement learning (RL) parameters that would affect decision-making processes in the lateral prefrontal cortex (LPFC). Previous neurophysiological experiments have shown MPFC activities encoding error likelihood, uncertainty, reward volatility, as well as neural responses categorizing different types of feedback, for instance, distinguishing between choice errors and execution errors. Rushworth and colleagues have proposed that the involvement of MPFC in tracking the volatility of the task could contribute to the regulation of one of RL parameters called the learning rate. We extend this hypothesis by proposing that MPFC could contribute to the regulation of other RL parameters such as the exploration rate and default action values in case of task shifts. Here, we analyze the sensitivity to RL parameters of behavioral performance in two monkey decision-making tasks, one with a deterministic reward schedule and the other with a stochastic one. We show that there exist optimal parameter values specific to each of these tasks, that need to be found for optimal performance and that are usually handtuned in computational models. In contrast, automatic online regulation of these parameters using some heuristics can help producing a good, although non-optimal, behavioral performance in each task. We finally describe our computational model of MPFC-LPFC interaction used for online regulation of the exploration rate and its application to a human-robot interaction scenario. There, unexpected uncertainties are produced by the human introducing cued task changes or by cheating. The model enables the robot to autonomously learn to reset exploration in response to such uncertain cues and events. The combined results provide concrete evidence specifying how prefrontal cortical subregions may cooperate to regulate RL parameters. It also shows how such neurophysiologically inspired mechanisms can control advanced robots in the real Progress in Brain Research, Volume 202, ISSN 0079-6123, http://dx.world. Finally, the model's learning mechanisms that were challenged in the last robotic scenario provide testable predictions on the way monkeys may learn the structure of the task during the pretraining phase of the previous laboratory experiments.

Download

Optic Flow Stimuli Update Anterodorsal Thalamus Head Direction Neuronal Activity in Rats

Journal of Neuroscience, 2013

Head direction (HD) neurons fire selectively according to head orientation in the yaw plane relat... more Head direction (HD) neurons fire selectively according to head orientation in the yaw plane relative to environmental landmark cues. Head movements provoke optic field flow signals that enter the vestibular nuclei, indicating head velocity, and hence angular displacements. To test whether optic field flow alone affects the directional firing of HD neurons, rats walked about on a circular platform as a spot array was projected onto the surrounding floor-to-ceiling cylindrical black curtain. Directional responses in the anterodorsal thalamus of four rats remained stable as they moved about with the point field but in the absence of landmark cues. Then, the spherical projector was rotated about its yaw axis at 4.5°/s for ϳ90 s. In 27 sessions the mean drift speed of the preferred directions (PDs) was 1.48°/s (SDϭ0.78°/s; range: 0.15 to 2.88°/s). Thus, optic flow stimulation entrained PDs, albeit at drift speeds slower than the field rotation. This could be due to conflicts with vestibular, motor command, and efferent copy signals. After field rotation ended, 20/27 PDs drifted back to within 45°of the initial values over several minutes, generally following the shortest path to return to the initial value. Poststimulation drifts could change speed and/or direction, with mean speeds of 0.68Ϯ0.64°/s (range 0 to 1.36°/s). Since the HD cell pathway (containing anterodorsal thalamus) is the only known projection of head direction information to entorhinal grid cells and hippocampal place cells, yaw plane optic flow signals likely influence representations in this spatial reference coordinate system for orientation and navigation.

Download

Modelling the learning of biomechanics and visual planning for decision-making of motor actions

Journal of Physiology-Paris, 2013

Recent experiments showed that the bio-mechanical ease and end-point stability associated to reac... more Recent experiments showed that the bio-mechanical ease and end-point stability associated to reaching movements are predicted prior to movement onset, and that these factors exert a significant influence on the choice of movement. As an extension of these results, here we investigate whether the knowledge about biomechanical costs and their influence on decision-making are the result of an adaptation process taking place during each experimental session or whether this knowledge was learned at an earlier stage of development. Specifically, we analysed both the pattern of decision-making and its fluctuations during each session, of several human subjects making free choices between two reaching movements that varied in path distance (target relative distance), biomechanical cost, aiming accuracy and stopping requirement. Our main result shows that the effect of biomechanics is well established at the start of the session, and that, consequently, the learning of biomechanical costs in decision-making occurred at an earlier stage of development. As a means to characterise the dynamics of this learning process, we also developed a model-based reinforcement learning model, which generates a possible account of how biomechanics may be incorporated into the motor plan to select between reaching movements. Results obtained in simulation showed that, after some pre-training corresponding to a motor babbling phase, the model can reproduce the subjects' overall movement preferences. Although preliminary, this supports that the knowledge about biomechanical costs may have been learned in this manner, and supports the hypothesis that the fluctuations observed in the subjects' behaviour may adapt in a similar fashion.

Download

Un modèle d'apprentissage par renforcement dans une architecture de contrôle de la sélection de l'action chez le rat artificiel Psikharpax

Optic field flow signals update the activity of head direction cells in the rat anterodorsal thalamus

JOURNAL OF VESTIBULAR RESEARCH., 2004

Complementary roles of the rat prefrontal cortex and striatum in reward-based learning and shifting navigation strategies

A computational model of integration between reinforcement learning and task monitoring in the prefrontal cortex

by Peter Dominey and Mehdi Khamassi

Taking inspiration from neural principles of decision-making is of particular interest to help im... more Taking inspiration from neural principles of decision-making is of particular interest to help improve adaptivity of artificial systems. Research at the crossroads of neuroscience and artificial intelligence in the last decade has helped understanding how the brain organizes reinforcement learning (RL) processes (the adaptation of decisions based on feedback from the environment). The current challenge is now to understand how the brain flexibly regulates parameters of RL such as the exploration rate based on the task structure, which is called metalearning ([1] : . Here, we propose a computational mechanism of exploration regulation based on real neurophysiological and behavioral data recorded in monkey prefrontal cortex during a visuo-motor task involving a clear distinction between exploratory and exploitative actions. We first fit trial-by-trial choices made by the monkeys with an analytical reinforcement learning model. We find that the model which has the highest likelihood of predicting monkeys' choices reveals different exploration rates at different task phases. In addition, the optimized model has a very high learning rate, and a reset of action values associated to a cue used in the task to signal condition changes. Beyond classical RL mechanisms, these results suggest that the monkey brain extracted task regularities to tune learning parameters in a task-appropriate way. We finally use these principles to develop a neural network model extending a previous cortico-striatal loop model. In our prefrontal cortex component, prediction error signals are extracted to produce feedback categorization signals. The latter are used to boost exploration after errors, and to attenuate it during exploitation, ensuring a lock on the currently rewarded choice. This model performs the task like monkeys, and provides a set of experimental predictions to be tested by future neurophysiological recordings.

Download

Behavioral Regulation and the Modulation of Information Coding in the Lateral Prefrontal and Cingulate Cortex

by Peter Dominey and Mehdi Khamassi

Cerebral Cortex, 2014

To explain the high level of flexibility in primate decision-making, theoretical models often inv... more

Download

Meta-Learning, Cognitive Control, and Physiological Interactions between Medial and Lateral Prefrontal Cortex

by Mehdi Khamassi, Marie Rothé, and Emmanuel Procyk

Neural Basis of Motivational and Cognitive Control, 2011

Download

Neuro-inspired navigation strategies shifting for robots: Integration of a multiple landmark taxon strategy

by Christophe Grand, Mehdi Khamassi, and Steve N'Guyen

Rodents have been widely studied for their adaptive navigation capabilities. They are able to exh... more Rodents have been widely studied for their adaptive navigation capabilities. They are able to exhibit multiple navigation strategies; some based on simple sensory-motor associations, while others rely on the construction of cognitive maps. We previously proposed a computational model of parallel learning processes during navigation which could reproduce in simulation a wide set of rat behavioral data and which could adaptively control a robot in a changing environment. In this previous robotic implementation the visual approach (or taxon) strategy was however paying attention to the intra-maze landmark only and learned to approach it. Here we replaced this mechanism by a more realistic one where the robot autonomously learns to select relevant landmarks. We show experimentally that the new taxon strategy is efficient, and that it combines robustly with the planning strategy, so as to choose the most efficient strategy given the available sensory information.

Download

Rat anterodorsal thalamic head direction neurons depend upon dynamic visual signals to select anchoring landmark cues

by E. Burguière, Mehdi Khamassi, Sidney Wiener, and Eric Burguiere

European Journal of Neuroscience, 2004

Head direction cells, which are functionally coupled to 'place' cells of the hippocampus, a struc... more Head direction cells, which are functionally coupled to 'place' cells of the hippocampus, a structure critically involved in spatial cognition, are likely neural substrates for the sense of direction. Here we studied the mechanism by which head direction cells are principally anchored to background visual cues [M.B. Zugaro et al. (2001) J. Neurosci., 21, RC154,1-5]. Anterodorsal thalamic head direction cells were recorded while the rat foraged on a small elevated platform in a 3-m diameter cylindrical enclosure. A large card was placed in the background, near the curtain, and a smaller card was placed in the foreground, near the platform. The cards were identically marked, proportionally dimensioned, subtended the same visual angles from the central vantage point and separated by 90°. The rat was then disoriented in darkness, the cards were rotated by 90°in opposite directions about the center and the rat was returned. Preferred directions followed either the background card, foreground card or midpoint between the two cards. In continuous lighting, preferred directions shifted to follow the background cue in most cases (30 of the 53 experiments, Batschelet V-test, P < 0.01). Stroboscopic illumination, which perturbs dynamic visual signals (e.g. motion parallax), blocked this selectivity. Head direction cells remained equally anchored to the background card, foreground card or configuration of the two cards (Watson test, P > 0.1). This shows that dynamic visual signals are critical in distinguishing typically more stable background cues which govern spatial neuronal responses and orientation behaviors.

Download

Mehdi Khamassi

Uploads

Papers by Mehdi Khamassi

Log In