Bounded recursive self-improvement

Journal of Artificial General Intelligence, 2013

Intuitively speaking, "self-programming" means the ability for a computer system to program its own actions. This notion is clearly related to Artificial Intelligence, and has been used by many researchers. Like many other high-level concepts, however, scrutiny shows that the term can be interpreted in several different ways. To make the discussion concrete and meaningful we introduce here a working definition of self-programming. In this definition we increase its concreteness while trying to keep the intuitive meaning of the concept. The activities of a computer system usually are considered to consist of atomic actions (which can also be called instructions, operations, behavior, or something else in different contexts). At any given moment the system's primitive actions are in a finite and constant set A, meaning that they are distinct from each other, and can be enumerated. An action may take some input arguments, and produce some output arguments. The system can execute each of its actions, This work is licensed under the Creative Commons Attribution 3.0 License.

Ai Magazine, 2007

To construct a perpetual self-aware cognitive agent that can continuously operate with independence, an introspective machine must be produced. To assemble such a agent, it is necessary to perform a full integration of cognition (planning, understanding, and learning) and metacognition (control and monitoring of cognition) with intelligent behaviors. The failure to do this completely is why similar more limited efforts have not succeeded in the past. As a start toward this goal, I performed an integration of an introspective multistrategy learning system with a nonlinear state-space planning agent using the wumpus world as environment. In this integration I show how the resultant system I call INTRO can generate its own goals. I use this system to discuss issues of self-awareness by machine.

presented and distributed at the, 2007

Self-improving systems are a promising new approach to developing artificial intelligence. But will their behavior be predictable? Can we be sure that they will behave as we intended even after many generations of selfimprovement? This paper presents a framework for answering questions like these. It shows that self-improvement causes systems to converge on an 2 architecture that arises from von Neumann's foundational work on microeconomics. Self-improvement causes systems to allocate their physical and computational resources according to a universal principle. It also causes systems to exhibit four natural drives: 1) efficiency, 2) self-preservation, 3) resource acquisition, and 4) creativity. Unbridled, these drives lead to both desirable and undesirable behaviors. The efficiency drive leads to algorithm optimization, data compression, atomically precise physical structures, reversible computation, adiabatic physical action, and the virtualization of the physical. It also governs a system's choice of memories, theorems, language, and logic. The self-preservation drive leads to defensive strategies such as "energy encryption" for hiding resources and promotes replication and game theoretic modeling. The resource acquisition drive leads to a variety of competitive behaviors and promotes rapid physical expansion and imperialism. The creativity drive leads to the development of new concepts, algorithms, theorems, devices, and processes. The best of these traits could usher in a new era of peace and prosperity; the worst are characteristic of human psychopaths and could bring widespread destruction. How can we ensure that this technology acts in alignment with our values? We have leverage both in designing the initial systems and in creating the social context within which they operate. But we must have clarity about the future we wish to create. We need not just a logical understanding of the technology but a deep sense of the values we cherish most. With both logic and inspiration we can work toward building a technology that empowers the human spirit rather than diminishing it.

Proceedings of the 36th European Conference on Cognitive Ergonomics, 2018

Recent progress in Artificial Intelligence, sensing and network technology, robotics, and (cloud) computing has enabled the development of intelligent autonomous machine systems. Telling such autonomous systems "what to do" in a responsible way, is a non-trivial task. For intelligent autonomous machines to function in human society and collaborate with humans, we see three challenges ahead affecting meaningful control of autonomous systems. First, autonomous machines are not yet capable of handling failures and unexpected situations. Providing procedures for all possible failures and situations is unfeasible because the state-action space would explode. Machines should therefore become self-aware (self-assessment, self-management) enabling them to handle unexpected situations when they arise. This is a challenge for the computer science community. Second, in order to keep (meaningful) control, humans come into a new role of providing intelligent autonomous machines with objectives or goal functions (including rules, norms, constraints and moral values), specifying the utility of every possible outcome of actions of autonomous machines. Third, in order to be able to collaborate with humans, autonomous systems will require an understanding of (us) humans (i.e., our social, cognitive, affective and physical behaviors) and the ability to engage in partnership interactions (such as explanations of task performances, and the establishment of joint goals and work agreements). These are new challenges for the cognitive ergonomics community. CCS CONCEPTS • Human-centered computing → Interaction design → Interaction design theory, concepts and paradigms;

Bootstrapping is a widely employed technique in the process of building highly complex systems such as microprocessors, language compilers, and computer operating systems. It could play an even more prominent role in the creation of computation systems capable of supporting intelligent agent behaviors because of the even higher level of complexity. The prospect of a self-bootstrapping, self-improving intelligent system has motivated various fields of research in machine learning. However, a robust, generalizable methodology of machine learning is yet to be found; there are still a lot of learning behaviors that no existing learning technique can adequately account for. We believe a uniform, logic-based system such as active logic [1, 2], will be more successful in the realization of this ideal. The overall architecture that we envision is as follows: a central commonsense reasoner module attends to novel situations where the system does not already have expertise, and to its own failures; it then reasons its way to solutions or repairs, and puts these into action while at the same time causing "expert" modules to be either created or retrained so as to more quickly enact those solutions on future occasions. Thus what we propose is a kind of meld between declarative and procedural techniques where the former has great expressive power and flexibility (but is slow) and the latter is very fast but hard to adapt to new situations. We will explore the possibilities of using reflection and continual computation towards this end.

Four principal features of autonomous control systems are left both unaddressed and unaddressable by present-day engineering methodologies: 1. The ability to operate effectively in environments that are only partially known beforehand at design time; 2. A level of generality that allows a system to re-assess and re-define the fulfillment of its mission in light of unexpected constraints or other unforeseen changes in the environment; 3. The ability to operate effectively in environments of significant complexity; and 4. The ability to degrade gracefully – how it can continue striving to achieve its main goals when resources become scarce, or in light of other expected or unexpected constraining factors that impede its progress. We describe new methodological and engineering principles for addressing these shortcomings, that we have used to design a machine that becomes increasingly better at behaving in underspecified circumstances, in a goal-directed way, on the job, by modeling itself and its environment as experience accumulates. Based on principles of autocatalysis, endogeny, and reflectivity, the work provides an architectural blueprint for constructing systems with high levels of operational autonomy in underspecified circumstances, starting from only a small amount of designer-specified code – a seed. Using a value- driven dynamic priority scheduling to control the parallel execution of a vast number of lines of reasoning, the system accumulates increasingly useful models of its experience, resulting in recursive self-improvement that can be autonomously sustained after the machine leaves the lab, within the boundaries imposed by its designers. A prototype system has been implemented and demonstrated to learn a complex real-world task – real-time multimodal dialogue with humans – by on-line observation. Our work presents solutions to several challenges that must be solved for achieving artificial general intelligence.

2014

To construct a perpetual self-aware cognitive agent that can continuously operate with independence, an introspective machine must be produced. To assemble such a agent, it is necessary to perform a full integration of cognition (plan-ning, understanding, and learning) and metacognition (con-trol and monitoring of cognition) with intelligent behaviors. The failure to do this completely is why similar more limited efforts have not succeeded in the past. As a start toward this goal, I performed an integration of an introspective multi-strategy learning system with a nonlinear state-space plan-ning agent using the wumpus world as environment. In this integration I show how the resultant system I call INTRO can generate its own goals. I use this system to discuss issues of self-awareness by machine.

Artificial Intelligence, 1998

motives for space exploration have inspired NASA to work toward the goal of establishing a virtual presence in space, through heterogeneous fleets of robotic explorers. Information technology, and Artificial Intelligence in particular, will play a central role in this endeavor by endowing these explorers with a form of computational intelligence that we call remote ayems. In this paper we describe the Remote Agent, a specific autonomous agent architecture based on the principles of model-based programming, on-board deduction and search, and goaldirected closed-loop commanding, that takes a significant step toward enabling this future. This architecture addresses the unique characteristics of the spacecraft domain that require highly reliable autonomous operations over long periods of time with tight deadlines, resource constraints, and concurrent activity among tightly coupled subsystems. The Remote Agent integrates constraintbased temporal planning and scheduling, robust multi-threaded execution, and model-based mode identification and reconfiguration. The demonstration of the integrated system as an on-board controller for Deep Space One, NASA's first New Millennium mission, is scheduled for a period of a week in mid 1999. The development of the Remote Agent also provided the opportunity to reassess some of AI's conventional wisdom about the challenges of implementing embedded systems, tractable reasoning, and knowledge representation. We discuss these issues, and our often contrary experiences, throughout the paper. 0 1998 Published by Elsevier Science B.V.

The Challenge of …, 2008

This book has provided various theoretical perspectives on anticipatory processes in natural and artificial cognitive systems. Advantages have been proposed and confirmed in various detailed case studies, which may have given the reader detailed insights into anticipatory processes and their importance in various cognitive systems tasks. To wrap up these advantages and give a concluding overview of various current anticipatory process advantages, this final chapter highlights a concise collection of precise success stories of anticipations in artificial cognitive systems. We survey fourteen case studies, which were developed during the EU project Min-dRACES 1 . In these studies, simulated or real robots were tested in different environmental tasks, which required advanced sensorimotor and cognitive abilities. These abilities included the initiation and control of goal-directed actions, the orientation of attention, finding and reaching goal locations, and performing mental experiments for action selection. All the studies have shown advantages of anticipatory mechanisms compared to reactive mechanisms in terms of increased robot autonomy and adaptivity. In some cases, anticipations even caused the development of new cognitive abilities, which were simply impossible without anticipatory mechanisms. For each case study, we indicate relevant associated publications, in which the interested reader may find further details on the relevant computational architectures, the involved anticipatory mechanisms, as well as on the analytical and quantitative results.