Models of Gaze Control for Manipulation Tasks

Biological Cybernetics, 1990

Multiple gaze control capabilities are coordinated using predictive methods. Internal models of state allow smooth interaction of multivariable, multirate controls. The problem of rapid gaze shift in a simulated robotic system illustrates several styles of control interaction.

Pattern Recognition Letters, 2006

This paper shows that sensory-motor coordination contributes to the performance of situated models on the high-level task of artificial gaze control for gender recognition in static natural images. To investigate the advantage of sensory-motor coordination, we compare a non-situated model of gaze control with a situated model. The non-situated model is incapable of sensory-motor coordination. It shifts the gaze according to a fixed set of locations, optimised by an evolutionary algorithm. The situated model determines gaze shifts on the basis of local inputs in a visual scene. An evolutionary algorithm optimises the model's gaze control policy. In the experiments performed, the situated model outperforms the non-situated model. By adopting a Bayesian framework, we show that the mechanism of sensory-motor coordination is the cause of this performance difference. The essence is that the mechanism maximises task-specific information in the observations over time, by establishing dependencies between multiple actions and observations.

2005

In the field of artificial intelligence, there is a considerable interest in the notion of sensory-motor coordination as an explanation for intelligent behaviour. However, there has been little research on sensory-motor coordination in tasks that go beyond low-level behavioural tasks. In this paper we show that sensory-motor coordination can also enhance performance on a high-level task: artificial gaze control for gender recognition in natural images. To investigate the advantage of sensory-motor coordination, we compare a non-situated model of gaze control (incapable of sensory-motor coordination) with a situated model of gaze control (capable of sensory-motor coordination). The non-situated model of gaze control shifts the gaze according to a fixed set of locations, optimised by an evolutionary algorithm. The situated model of gaze control determines gaze shifts on the basis of local inputs in a visual scene. An evolutionary algorithm optimises the model’s gaze control policy. From the experiments performed, we may conclude that sensory-motor coordination contributes to artificial gaze control for the high-level task of gender recognition in natural images: the situated model outperforms the non-situated model. The mechanism of sensory-motor coordination establishes dependencies between multiple actions and observations that are exploited to optimise categorisation performance.

2017

The capability of directing gaze to relevant parts in the environment is crucial for our survival. Computational models based on ideal-observer theory have provided quantitative accounts of human gaze selection in a range of visual search tasks. According to these models, gaze is directed to the position in a visual scene, at which uncertainty about task relevant properties will be reduced maximally with the next look. However, in tasks going beyond a single action, delayed rewards can play a crucial role thereby necessitating planning. Here we investigate whether humans are capable of planning more than the next single eye movement. We found evidence that our subjects’ behavior was better explained by an ideal planner compared to the ideal observer. In particular, the location of the first fixation differed depending on the stimulus and the time available for the search. Overall, our results are the first evidence that our visual system is capable of planning.

Visual Cognition, 2009

Gaze changes and the resultant fixations that orchestrate the sequential acquisition of information from the visual environment are the central feature of primate vision. How are we to understand their function? For the most part, theories of fixation targets have been image based: The hypothesis being that the eye is drawn to places in the scene that contain discontinuities in image features such as motion, colour, and texture. But are these features the cause of the fixations, or merely the result of fixations that have been planned to serve some visual function? This paper examines the issue and reviews evidence from various image-based and task-based sources. Our conclusion is that the evidence is overwhelmingly in favour of fixation control being essentially task based.

Current Biology, 2014

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021

Scientific Reports

People must decide where, when, and for how long to allocate gaze to perform different motor behaviours. However, the factors guiding gaze during these ongoing, natural behaviours are poorly understood. Gaze shifts help acquire information, suggesting that people should direct gaze to locations where environmental details most relevant to the task are uncertain. To explore this, human subjects stepped on a series of targets as they walked. We used different levels of target uncertainty, and through instruction, altered the importance of (or subjective value assigned to) foot-placement accuracy. Gaze time on targets increased with greater target uncertainty when precise foot placement was more important, and these longer gaze times associated with reduced foot-placement error. Gaze times as well as the gaze shifts to and from targets relative to stepping differed depending on the target's position in the sequence and uncertainty level. Overall, we show that gaze is allocated to reduce uncertainty about target locations, and this depends on the value of this information gain for successful task performance. Furthermore, we show that the spatial-temporal pattern of gaze to resolve uncertainty changes with the evolution of the motor behaviour, indicating a flexible strategy to plan and control movement. To acquire environmental details necessary for performing a visually guided action, such as locating a landmark, reaching to grasp a glass, avoiding obstacles, and regulating foot placement, appropriate temporal and spatial gaze shifts are required. Consider the situation where you are hiking in the woods; you must identify hazards and obstacles, choose the route you wish to take, and step to desired locations on the ground. Here the decision where, when, and for how long to look has important implications for safety, and thus the coupling between gaze location and foot placement is critical. Although novel stimuli and image salience can capture attention and direct gaze, as supported by computer-based visual tasks and computational models 1 , recent research shows that gaze fixations during more naturalistic behaviours are highly task-relevant 2-9. For instance, when making a sandwich, eye movements are directed to the knife, the jelly jar, the bread, and the plate before each item is manipulated 2. When walking across difficult terrain, people predominantly fixate where they will eventually step 5. However, there is little understanding of how or why task-relevant locations are selected and prioritized, or what determines how much time a location is fixated. Consequently, a central unanswered question emerges: what factors determine how gaze is allocated in visually guided motor behaviours? Several brain regions implicated in the control of eye movements are sensitive to reward probability 10-13. For example, the discharge activity of neurons within the monkey lateral intraparietal area (LIP) varies according to the expected (juice) reward associated with an eye movement to a visual target 13. With walking and other motor actions outside the lab, however, fixating a location does not usually elicit a reward. Rather, gaze shifts help the brain gather relevant details necessary for making a motor decision, such as where to place the foot. Thus, reward alone cannot explain gaze allocation during ongoing, naturalistic behaviours. Interestingly, Foley et al. 14 recently showed that certain LIP neurons change firing rates depending on the expected gain in information needed to perform the second action in a two-step decision task, rather than for the expected reward associated with that subsequent action. This highlights the importance of immediate information gain in shaping action decisions. Since our knowledge of the world is imperfect, our sensory feedback is noisy, and the environment changes as movement unfolds over time, this means that many environmental features relevant to an action are uncertain.

IEEE Transactions on Neural Systems and Rehabilitation Engineering

Robots capable of robust, real-time recognition of human intent during manipulation tasks could be used to enhance human-robot collaboration for activities of daily living. Eye gaze-based control interfaces offer a non-invasive way to infer intent and reduce the cognitive burden on operators of complex robots. Eye gaze is traditionally used for "gaze triggering" (GT) in which staring at an object, or sequence of objects, triggers pre-programmed robotic movements. We propose an alternative approach: a neural network-based "action prediction" (AP) mode that extracts gaze-related features to recognize, and often predict, an operator's intended action primitives. We integrated the AP mode into a shared autonomy framework capable of 3D gaze reconstruction, real-time intent inference, object localization, obstacle avoidance, and dynamic trajectory planning. Using this framework, we conducted a user study to directly compare the performance of the GT and AP modes using traditional subjective performance metrics, such as Likert scales, as well as novel objective performance metrics, such as the delay of recognition. Statistical analyses suggested that the AP mode resulted in more seamless robotic movement than the state-of-the-art GT mode, and that participants generally preferred the AP mode.

Understanding others' actions as goal-directed is a key mechanism to develop social cognitive abilities such as imitation and cooperation. Recent findings in psychology have demonstrated that the ability to predict the goal of observed actions emerges as early as six months in infancy. However, what mechanisms are involved and how they trigger the development of this ability are still open questions. In this paper, we propose a computational model employing a recurrent neural network to reproduce the developmental process of goal-directed gaze shift. Our hypothesis is that it is possible to infer the goal of observed actions by learning to predict the attention target originated from bottom-up visual attention. While keeping consistency with psychological findings, our experimental results confirmed the hypothesis that learning to predict the attention targets leads to the development of predictive gaze shift to the action goal.