Learning unknown event models

2001

Mainly rooted in the cognitive-psychoevolutionary model of surprise proposed by the research group of the University of Bielefeld (Meyer, Reisenzein, Schützwohl, etc.), the computational model of surprise described in this paper relies on the assumption that surprise-eliciting events initiate a series of mental processes that begin with the appraisal of unexpectedness, continue with the interruption of ongoing activity and the focusing of attention on the unexpected event, and end with the analysis and evaluation of that event plus revision of beliefs. With respect to the computation of unexpectedness, the model also incorporates suggestions by Ortony and Partridge. This model of surprise is implemented in an artificial agent called S-EUNE, whose task is to explore uncertain and unknown environments. The accuracy of our surprise model was evaluated in a series of experimental tests that focused on the comparison of surprise intensity values generated by the artificial agent with rat...

Proceedings of the AAAI Conference on Artificial Intelligence, 2019

In part motivated by topics such as agency safety, there is an increasing interest in goal reasoning, a form of agency where the agents formulate their own goals. One of the crucial aspects of goal reasoning agents is their ability to detect if the execution of their courses of actions meet their own expectations. We present a taxonomy of different forms of expectations as used by goal reasoning agents when monitoring their own execution. We summarize and contrast the current understanding of how to define and check expectations based on different knowledge sources used. We also identify gaps in our understanding of expectations.

2007

Abstract—Situated agents engaged in open systems continually face with external events requiring adequate services and behavioral responses. In these conditions agents should be able to improve their adaptivity over time, namely 1) to deal with and anticipate relevant changes and critical situations, 2) to temporally define relative priorities between goals varying their importance over time and 3) to use informational feedback to learn from experience and become better at achieving their goals.

Intelligent virtual agents, 2003

In this paper we describe the implementation of interactive agents capable of gathering and extending their knowledge. Interactive agents are designed to perform tasks requested by a user in natural language. Using simple sentences the agent can answer questions and in case a task can not be fulfilled the agent must communicate with the user. In particular, an interactive agent can tell when necessary information for a task is missing, giving the user a chance to supply this information, which may in effect result in teaching the agent. The ...

FRONTIERS IN ARTIFICIAL …, 2006

In this work we introduce some anticipatory mechanisms in a BDI architecture, and model part of the functions of surprise, in particular the advantage of becoming cautious, wary and careful. This approach increases the opportunism and the reactivity of BDIlike planning and deliberative agents. The relevance of this issue will soon become important both for software cognitive agents and for autonomous robots. To have anticipatory representations, to experience surprise, and to exploit various functions of them will be crucial, not only for curiosity and exploration, but much more for learning, attention capabilities, for prudently adjusting behaviours, and for social relations like trust and suspicion, coordination and reliance etc. We provide in this paper a first modeling of those kind of agents and we propose a general BDI architecture to allow caution and expectation enabled agents. Finally we provide experiments illustrating different performances for agents designed with different caution policies.

2018

Complex, real-world environments may not be fully modeled for an agent, especially if the agent has never operated in the environment before. The agent’s ability to effectively plan and act in the environment is influenced by its knowledge of when it can perform specific actions and the effects of those actions. We describe progress on an explainable exploratory planning agent that is capable of learning action preconditions using an exploratory planning process. The agent’s architecture allows it to perform both exploratory actions as well as goal-directed actions, which opens up important considerations for how exploratory planning and goal planning should be controlled, as well as how the agent’s behavior should be explained to any teammates it may have. We describe initial approaches for both exploratory planning and action model learning, and evaluate them in the video game Dungeon Crawl Stone Soup.

2016

Goal Driven Autonomy (GDA) is an agent model for reasoning about goals while acting in a dynamic environment. Since anomalous events may cause an agent's current goal to become invalid, GDA agents monitor the environment for such anomalies. When domains are both partially observable and dynamic, agents must reason about sensing and planning actions. Previous GDA work evaluated agents in domains that were partially observable, but does not address sensing actions with associated costs. Furthermore, partial observability still enabled generation of a grounded plan to reach the goal. We study agents where observability is more limited: the agent cannot generate a grounded plan because it does not know which future actions will be available until it explores more of the environment. We present a formalism of the problem that includes sensing costs, a GDA algorithm using this formalism, an examination of four methods of expectations under this formalism, and an implementation of the ...

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, 2020

We consider an agent that operates with two models of the environment: one that captures expected behaviors and one that captures additional exceptional behaviors. We study the problem of synthesizing agent strategies that enforce a goal against environments operating as expected while also making a best effort against exceptional environment behaviors. We formalize these concepts in the context of linear-temporal logic, and give an algorithm for solving this problem. We also show that there is no trade-off between enforcing the goal under the expected environment specification and making a best-effort for it under the exceptional one.

ArXiv, 2020

We present a novel form of explanation for Reinforcement Learning, based around the notion of intended outcome. These explanations describe the outcome an agent is trying to achieve by its actions. We provide a simple proof that general methods for post-hoc explanations of this nature are impossible in traditional reinforcement learning. Rather, the information needed for the explanations must be collected in conjunction with training the agent. We derive approaches designed to extract local explanations based on intention for several variants of Q-function approximation and prove consistency between the explanations and the Q-values learned. We demonstrate our method on multiple reinforcement learning problems, and provide code to help researchers introspecting their RL environments and algorithms.

Cognitive Systems Research, 2012

We describe a Belief-Desire-Intention-like architecture for an explorer agent in which the psychological constructs of surprise and curiosity play an important role in decision-making, particularly in the selection of view-points during the process of exploring unknown environments. Taking into account previous studies about the psychological constructs involved in exploratory behaviour, the agent is equipped in advance with the basic desires for maximal information gain (reduce curiosity), and maximal surprise. However, to reflect Berlyne's theory that says that the tendency to explore the environment occurs in the absence of known drives, we considered also the basic desire for minimal hunger as a representative example of those additional basic desires that can restrain exploration. This surprisecuriosity-based exploration strategy was confronted with a "cold" classical exploration strategy in environments populated with entities. The results of this experiment indicate that the classical strategy outperforms slightly the surprise-curiosity-based one with respect to the exploration performance measures of the time/energy required to explore all the environment completely, and the time/energy required to explore all the entities. However, the classical strategy was outperformed by the surprise-curiosity-based one with respect to the time/ energy required to explore all different entities, and consequently, with more evidence, with respect to the number of steps (trips between two entities) required to explore all different entities. This is a valuable result for resource-bounded, active learning agents that benefit from choosing the more informative data from which they learn while ignoring time-consuming/expensive, redundant data. This important result is confirmed by the results of the analysis of the agents' behaviour exhibited along the traversing paths in the environment. The experiment also provided results concerning the robustness of the surprise-curiosity-based approach by assessing the influence of surprise and curiosity in several environments of different complexity and with different amplitudes for the visual field of the agent.