Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Summary of Foundations
Self Verification Results
Results
Limitations of Neat
Previous Work on Limitations
Reasons for the Results
Future Work
References
All Topics
Computer Science

Recurrence and Plasticity in Evolved Neural Controllers

Profile image of Mark AhlstromMark Ahlstrom

2009

Abstract

Mark Ahlstrom. RECURRENCE AND PLASTICITY IN EVOLVED ADAPTIVE NEURAL CONTROLLERS. (Under the direction of Dr. M. H. N. Tabrizi). Department of Computer Science, December 10 2009. Among the more important applications of evolutionary neurocontrollers is the development of systems that are able to dynamically adapt to a changing environment. While traditional approaches to control system design demand that the developer attempt to foresee all possible situations within which the controller may operate, neuroevolutionary approaches can facilitate the design of systems that are capable of operating in unforeseen circumstances. This paper examines two methods that have been used to provide for this adaptivity. The first method is the use of recurrent neural networks that have fixed connection weights. The second develops neurocontrollers with plastic synapses, thus allowing for the adaptation of the connection weights. Previous experimental results have shown that while both approaches can facilitate adaptive behavior, neural plasticity does not necessarily confer the expected benefits. In experiments using the NeuroEvolution of Augmenting Topologies (NEAT) method, discovered that in simple cases, recurrence was sufficient in solving at least some control problems. I examined whether or not these initial results continue to scale upwards into more complex problem spaces. This was done through a series of experiments ranging from controlling a simplified cannon shot to attempting to evolve neural flight controllers capable of flying different airplanes through a series of waypoints. The results of these experiments indicate that the NEAT algorithm itself is unable to scale efficiently to some larger problem spaces.

Related papers

Evolution of neurocontrollers for complex systems: alternatives to the incremental approach
Stéphane Doncieux

2004

Abstract Applications of neural networks to solve challenging control problems still face important difficulties when scalability issues are involved. Usually, such difficulties are tackled according to an incremental approach that consists in decomposing a given task into simpler sub-tasks that may be separately solved. In this article, we describe two alternatives to this approach and demonstrate their efficiency on the control of a simulated lenticular blimp, ie, a complex dynamic platform with five sensors and seven motors.

View PDFchevron_right
Evolving a Robot Neurocontroller for Memorization Tasks
Sule Yildirim-Yayilgan

In this study, a robotic neurocontroller architecture that handles delayed response tasks, which are characterized by a significant (and typically varying) delay between a stimulus and the corresponding appropriate response, is proposed. That is, the architecture aims at memorization and counting tasks. Current controller architectures use supervised (eg. gradient descent), reinforcement learning or evolutionary algorithms for this class of learning tasks. However, gradient descent methods have serious limitations such as being unable to handle memory very well, whereas supervised and reinforcement learning require a priori knowledge which might not always be available. Approaches for evolution of ANN as a controller includes evolution of initial weights, evolution of learning rules and evolving fixed weights. In this paper, initial and fixed weights are used for evolution of response and delay tasks.

View PDFchevron_right
Evolution of Neural Architecture Fitting Environmental Dynamics Evolution of Neural Architecture Fitting Environmental Dynamics
Genci Capi

Temporal and sequential information is essential to any agent continually interacting with its environment. In this paper, we test whether it is possible to evolve a recurrent neural network controller to match the dynamic requirement of the task. As a benchmark, we consider a sequential navigation task where the agent has to alternately visit two rewarding sites to obtain food and water after first visiting the nest. To achieve a better fitness, the agent must select relevant sensory inputs and update its working memory to realize a non-Markovian sequential behavior in which the preceding state alone does not determine the next action. We compare the performance of a feed-forward and recurrent neural control architectures in different environment settings and analyze the neural mechanisms and environment features exploited by the agents to achieve their goal. Simulation and experimental results using the Cyber Rodent robot show that a modular architecture with a locally excitatory recurrent layer outperformed the general recurrent controller. Keywords non-Markovian sequential task · recurrent neural network · evolutionary algorithm · evolved behavior

View PDFchevron_right
Evolved Neurodynamics for Robot Control
Frank Pasemann

Small recurrent neural network with two and three neurons are able to control autonomous robots showing obstacle avoidance and photo-tropic behaviors. They have been generated by evolutionary processes, and they demonstrate, how dynamical properties can be used for an effective behavior control. Presented examples also show how sensor fusion can be obtained by evolution. Additional techniques are used to excavate the relevant neural processing mechanisms underlying specific behavior features.

View PDFchevron_right
Evolving a Neurocontroller Through a Process of
Diego Federici

We introduce a model of cellular growth that generates neurocontrollers capable of guiding simple simulated agents in a harvesting task.

View PDFchevron_right
Artificial Evolution of Plastic Neural Networks: a few Key Concepts
J.-B. Mouret

2011

Abstract—This paper introduces a hierarchy of concepts to classify the goals and the methods of works that mix neuroevolution and synaptic plasticity. We propose definitions of “behavioral robustness” and oppose it to “reward-based behavioral changes”; we then distinguish the switch between behaviors and the acquisition of new behaviors. Last, we summarize the concept of “synaptic General Learning Abilities”(sGLA) and that of “synaptic Transitive learning Abilities (sTLA).

View PDFchevron_right
Adaptive Genomic Evolution of Neural Network Topologies (AGENT) for State-to-Action Mapping in Autonomous Agents
Amir Behjat

2019 International Conference on Robotics and Automation (ICRA), 2019

Neuroevolution is a process of training neural networks (NN) through an evolutionary algorithm, usually to serve as a state-to-action mapping model in control or reinforcement learning-type problems. This paper builds on the Neuro Evolution of Augmented Topologies (NEAT) formalism that allows designing topology and weight evolving NNs. Fundamental advancements are made to the neuroevolution process to address premature stagnation and convergence issues, central among which is the incorporation of automated mechanisms to control the population diversity and average fitness improvement within the neuroevolution process. Insights into the performance and efficiency of the new algorithm is obtained by evaluating it on three benchmark problems from the Open AI platform and an Unmanned Aerial Vehicle (UAV) collision avoidance problem. the dimension of the problem for many cases [7]. Most RL variants (with recent exceptions [8], [10]) are also not conducive to application in the continuous-space domain. An alternative class of frameworks, based on evolutionary algorithms [9], namely neuroevolution [13] and evolution strategies [14], seek to mitigate these limitations.

View PDFchevron_right
Evolution of a Subsumption Architecture Neurocontroller
Julian Togelius

Computing Research Repository, 2004

An approach to robotics called layered evolution and merging features from the subsumption architecture into evolutionary robotics is presented, and its advantages are discussed. This approach is used to construct a layered controller for a simulated robot that learns which light source to approach in an environment with obstacles. The evolvability and performance of layered evolution on this task is compared to (standard) monolithic evolution, incremental and modularised evolution. To corroborate the hypothesis that a layered controller performs at least as well as an integrated one, the evolved layers are merged back into a single network. On the grounds of the test results, it is argued that layered evolution provides a superior approach for many tasks, and it is suggested that this approach may be the key to scaling up evolutionary robotics.

View PDFchevron_right
Evolving Artificial Neural Networks through L-System and Evolutionary Computation (to be published at IJCNN 2015)
Mauro Roisenberg

The aim of this paper is to present a biologically inspired Neuro Evolutive Algorithm (NEA) able to generate modular, hierarchical and recurrent neural structures as those often found in the nervous system of live beings, and that enable them to solve intricate survival problems. In our approach we consider that a nervous system design and organization is a constructive process carried out by genetic information encoded in DNA. Our NEA evolves Artificial Neural Networks (ANNs) using a Lindenmayer System with memory that implements the principles of organization, modularity, repetition (multiple use of the same sub-structure), hierarchy (recursive composition of sub-structures), as a metaphor for development of neurons and its connections. In our method, this basic neural codification is integrated to a Genetic Algorithm (GA) that implements the constructive approach found in the evolutionary process, making it closest to the biological ones. Our method was initially tested on a simp...

View PDFchevron_right
Evolution of recurrent neural controllers using an extended parallel genetic algorithm
Genci Capi

Robotics and Autonomous Systems, 2005

Autonomous intelligent agents often must complete non-Markovian sequential tasks, which require complex recurrent neural controllers. In order to improve the convergence of evolution and reduce the computation time, this paper proposes application of an extended evolutionary algorithm. We implemented an extended multi-population genetic algorithm (EMPGA), where subpopulations apply different evolutionary strategies. In addition, subpopulations compete and cooperate among each other. Results show that EMPGA outperformed single population genetic algorithm (SPGA) by efficiently distributing the number of individuals among subpopulations as different strategies became successful during the course of evolution. In addition, the comparison with other multi-population GA shows that competition between subpopulations improved the quality of solution. The evolved neural controllers were also tested in the real hardware of Cyber Rodent robot.

View PDFchevron_right

References (97)

  1. 1 Diagram of the five parts of an artificial neuron. . . . . . . . . . . . .
  2. 2 Hebbian weight update of a neuron. . . . . . . . . . . . . . . . . . . .
  3. 3 Default Sigmoid activation function curve . . . . . . . . . . . . . . .
  4. 4 Sigmoid activation function with large threshold . . . . . . . . . . . .
  5. 5 Single Layer Neural Network . . . . . . . . . . . . . . . . . . . . . . .
  6. 6 Multi Layer Neural Network . . . . . . . . . . . . . . . . . . . . . . .
  7. 7 Recurrent Neural Network . . . . . . . . . . . . . . . . . . . . . . . .
  8. Nonstandard Neural Network . . . . . . . . . . . . . . . . . . . . . .
  9. 9 Nonstandard Functionality in a full network . . . . . . . . . . . . . .
  10. Kohonen SOM for character recognition. . . . . . . . . . . . . . . . .
  11. 11 Reinforcement control . . . . . . . . . . . . . . . . . . . . . . . . . .
  12. 12 Knapsack Problem encoding . . . . . . . . . . . . . . . . . . . . . . .
  13. 13 Initial Fitness Distribution . . . . . . . . . . . . . . . . . . . . . . . .
  14. 14 Knapsack Problem Initial Population . . . . . . . . . . . . . . . . . .
  15. 15 Knapsack Problem Initial Fitness . . . . . . . . . . . . . . . . . . . .
  16. 16 Fitness with poor selection . . . . . . . . . . . . . . . . . . . . . . . .
  17. 17 Single Point Crossover . . . . . . . . . . . . . . . . . . . . . . . . . .
  18. 18 Double Point Crossover . . . . . . . . . . . . . . . . . . . . . . . . . .
  19. 19 Cut and Splice Crossover . . . . . . . . . . . . . . . . . . . . . . . . .
  20. Competing Conventions . . . . . . . . . . . . . . . . . . . . . . . . .
  21. 1 Cannon average fitness . . . . . . . . . . . . . . . . . . . . . . . . . .
  22. 2 Cannon average nodes . . . . . . . . . . . . . . . . . . . . . . . . . .
  23. 3 Cannon best run fitness . . . . . . . . . . . . . . . . . . . . . . . . .
  24. 4 Cannon best run number nodes . . . . . . . . . . . . . . . . . . . . .
  25. 5 Cannon average adaptive fitness . . . . . . . . . . . . . . . . . . . . .
  26. 6 Cannon average number adaptive nodes . . . . . . . . . . . . . . . . .
  27. 7 Cannon best adaptive run fitness . . . . . . . . . . . . . . . . . . . .
  28. 8 Cannon best adaptive run number nodes . . . . . . . . . . . . . . . .
  29. 9 Robot for food foraging . . . . . . . . . . . . . . . . . . . . . . . . . .
  30. 10 Food foraging world . . . . . . . . . . . . . . . . . . . . . . . . . . . .
  31. 11 Food Foraging fitness . . . . . . . . . . . . . . . . . . . . . . . . . . .
  32. 12 Food Foraging average number nodes . . . . . . . . . . . . . . . . . .
  33. 13 Food Foraging best run fitness . . . . . . . . . . . . . . . . . . . . . .
  34. 14 Food Foraging best run number nodes . . . . . . . . . . . . . . . . .
  35. 15 Dangerous Food Foraging fitness . . . . . . . . . . . . . . . . . . . . .
  36. 16 Dangerous Food Foraging average number nodes . . . . . . . . . . . .
  37. 17 Dangerous Food Foraging best run fitness . . . . . . . . . . . . . . .
  38. 18 Dangerous Food Foraging best run number nodes . . . . . . . . . . .
  39. 19 Dangerous Food Foraging adaptive fitness . . . . . . . . . . . . . . .
  40. 20 Dangerous Food Foraging average adaptive number nodes . . . . . . .
  41. 21 Dangerous Food Foraging best adaptive run fitness . . . . . . . . . .
  42. 22 Dangerous Food Foraging best adaptive run number nodes . . . . . .
  43. 23 Safe vs Dangerous foraging fitness comparison . . . . . . . . . . . . .
  44. 24 Safe vs Dangerous foraging node comparison . . . . . . . . . . . . . .
  45. 1 Neural Flight Controller ANN . . . . . . . . . . . . . . . . . . . . . .
  46. 1 Recurrent Flight Controller average fitness . . . . . . . . . . . . . . .
  47. 2 Recurrent Flight Controller average number nodes . . . . . . . . . . .
  48. 3 Recurrent Flight Controller best run fitness . . . . . . . . . . . . . .
  49. 4 Recurrent Flight Controller best run number nodes . . . . . . . . . .
  50. 5 Flight Controller average fitness . . . . . . . . . . . . . . . . . . . . .
  51. 6 Flight Controller average number nodes . . . . . . . . . . . . . . . . .
  52. 7 Flight Controller best adaptive run fitness . . . . . . . . . . . . . . .
  53. 8 Flight Controller best adaptive run number nodes . . . . . . . . . . .
  54. 9 Single Plane Simplified Average Fitness . . . . . . . . . . . . . . . . .
  55. 10 Single Plane Simplified Average number nodes . . . . . . . . . . . . . BIBLIOGRAPHY
  56. Attik, M., Bougrain, L., and Alexandre, F. (2005). Neural network topology opti- mization. Lecture Notes in Computer Science, 3697:53.
  57. Auer, P., Burgsteiner, H., and Maass, W. (2007). A learning rule for very simple universal approximators consisting of a single layer of perceptrons. Neural Networks.
  58. Berndt, J. (2004). Jsbsim: An open source flight dynamics model in c++. AIAA Modeling and Simulation Technologies Conference and Exhibit.
  59. Beyeler, A., Zufferey, J., and Floreano, D. (2009). Vision-based control of near- obstacle flight. Autonomous Robots, pages 1-19.
  60. Chalmers, D. J. (1990). The evolution of learning: An experiment in genetic con- nectionism.
  61. Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Math- ematics of Control, Signals, and Systems (MCSS), 2(4):303-314.
  62. Daqi, G. and Shouyi, W. (1998). An optimization method for the topological struc- tures of feed-forward multi-layer neural networks. Pattern recognition, 31(9):1337- 1342.
  63. de Castro, L. (2007). Fundamentals of natural computing: an overview. Physics of Life Reviews, 4(1):1-36.
  64. Emmert-Streib, F. (2006). Influence of the neural network topology on the learning dynamics. Neurocomputing, 69(10-12):1179-1182.
  65. Ferrari, S. and Stengel, R. (2004). Online adaptive critic flight control. Journal of Guidance Control and Dynamics, 27:777-786.
  66. Floreano, D. (1998). Evolutionary re-adaptation of neurocontrollers in changing environments.
  67. Floreano, D. and Nolfi, S. (1997). Adaptive behavior in competing co-evolving species.
  68. Floreano, D. and Urzelai, J. (1999). Evolution of neural controllers with adaptive synapses and compact genetic encoding. Lecture Notes in Computer Science, pages 183-194.
  69. Ge, S., Hang, C., and Zhang, T. (1997). Direct adaptive neural network control of nonlinear systems. American Control Conference, 1997. Proceedings of the 1997, 3.
  70. Gomez, F. and Miikkulainen, R. (2003). Active guidance for a finless rocket using neuroevolution. Lecture Notes in Computer Science, pages 2084-2095.
  71. Hagan, M. and Demuth, H. (1999). Neural networks for control. American Control Conference, 1999. Proceedings of the 1999, 3.
  72. Jansen, T. (2001). On classifications of fitness functions. pages 371-385.
  73. Keane, A. J. (2001). An introduction to evolutionary computing in design search and optimisation. pages 1-11.
  74. Kirkpatrick, S., Gelatt, C., and Vecchi, M. (1983). Optimization by simulated an- nealing. Science, 220:671-680.
  75. Kohl, N. and Miikkulainen, R. (2008). Evolving neural networks for fractured do- mains. GECCO '08: Proceedings of the 10th annual conference on Genetic and evolutionary computation.
  76. Konen, W. and Bartz-Beielstein, T. (2009). Reinforcement learning for games: fail- ures and successes. Proceedings of the 11th annual conference companion . . . .
  77. Linhardt, M. and Butz, M. (2009). Neat in increasingly non-linear control situations. GECCO '09: Proceedings of the 11th annual conference companion on Genetic and evolutionary computation conference.
  78. Luger, G. F. (2001). Artificial Intelligence: Structures and Strategies for Complex Problem Solving. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.
  79. Minsky, M. and Papert, S. (1988). Perceptrons. pages 157-169.
  80. Monroy, G., Stanley, K., and Miikkulainen, R. (2006). Coevolution of neural net- works using a layered pareto archive. Proceedings of the 8th annual conference on Genetic and evolutionary computation, pages 329-336.
  81. Moriarty, D. E. and Miikkulainen, R. (1998). Forming neural networks through efficient and adaptive coevolution.
  82. Noriega, J. and Wang, H. (1995). A direct adaptive neural network control for unknown nonlinearsystems and its application. American Control Conference, 1995. Proceedings of the, 6.
  83. Padhy, N. (2007). Artificial Intelligence and Intelligent Systems. Oxford University Press, New Delhi, India.
  84. Pallett, T. and Ahmad, S. (1993). Adaptive neural network control of a helicopter in vertical flight. Aerospace Control Systems, 1993. Proceedings. The First IEEE Regional Conference on, pages 264-268.
  85. Pardoe, D., Ryoo, M., and Miikkulainen, R. (2005). Evolving neural network en- sembles for control problems. Proceedings of the 2005 conference on Genetic and evolutionary computation, pages 1379-1384.
  86. Pearlmutter, B. A. (1996). Gradient calculations for dynamic recurrent neural net- works: A survey.
  87. Principe, J. C., Euliano, N. R., and Lefebvre, W. C. (1999). Neural and Adaptive Systems: Fundamentals through Simulations with CD-ROM. John Wiley & Sons, Inc., New York, NY, USA.
  88. Reisinger, J. and Miikkulainen, R. (2007). Acquiring evolvability through adaptive representations. GECCO '07: Proceedings of the 9th annual conference on Genetic and evolutionary computation.
  89. Risi, S., Vanderbleek, S., Hughes, C., and Stanley, K. (2009). How novelty search escapes the deceptive trap of learning to learn. GECCO '09: Proceedings of the 11th Annual conference on Genetic and evolutionary computation.
  90. Rojas, R. (1996). Neural networks: A systematic introduction.
  91. Rowe, J. E. (2001). The dynamical systems model of the simple genetic algorithm. pages 31-57.
  92. Soltoggio, A. and Jones, B. (2009). Novelty of behaviour as a basis for the neuro- evolution of operant reward learning. GECCO '09: Proceedings of the 11th Annual conference on Genetic and evolutionary computation.
  93. Stanley, K. (2004). Efficient evolution of neural networks through complexification.
  94. Stanley, K., Bryant, B., and Miikkulainen, R. (2003). Evolving adaptive neural networks with and without adaptive synapses. Evolutionary Computation, 2003. CEC'03. The 2003 Congress on, 4.
  95. Stanley, K. and Miikkulainen, R. (2002). Evolving neural networks through aug- menting topologies. Evolutionary Computation, 10(2):99-127.
  96. Thomson, J., Jha, R., and Pradeep, S. (2004). Neurocontroller design for nonlinear control of takeoff of unmanned aerospace vehicles. Proceedings of the 42nd AIAA Aerospace Sciences Meeting and Exhibit, Reno, NV.
  97. Urzelai, J. and Floreano, D. (2001). Evolution of adaptive synapses: Robots with fast adaptive behavior in new environments. Evolutionary Computation, 9(4):495- 524. Valsalam, V. and Miikkulainen, R. (2009). Evolving symmetric and modular neu- ral networks for distributed control. GECCO '09: Proceedings of the 11th Annual conference on Genetic and evolutionary computation.

Related papers

Incremental Evolution of Neural Network Architectures for Adaptive Behaviour
Phil Husbands

1992

This paper describes aspects of our ongoing work in evolving recurrent dynamicalartificial neural networks which act as sensory-motor controllers, generating adaptivebehaviour in artificial agents. We start with a discussion of the rationale for ourapproach. Our approach involves the use of recurrent networks of artificial neuronswith rich dynamics, resilience to noise (both internal and external); and separateexcitation and inhibition channels. The

View PDFchevron_right
Evolving structure and function of neurocontrollers
Frank Pasemann

Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406), 1999

The presented evolutionary algorithm is especially designed to generate recurrent neural networks with non-trivial internal dynamics. It is not based on genetic algorithms, and sets no constraints on the number of neurons and the architecture of a network. Network topology and parameters like synaptic weights and bias terms are developed simultaneously. It is well suited for generating neuromodules acting in sensorimotor loops, and therefore it can be used for evolution of neurocontrollers solving also nonlinear control problems. We demonstrate this capability by applying the algorithm successfully to the following task: A rotating pendulum is mounted on a cart; stabilize the rotator in an upright position, and center the cart in a given finite interval.

View PDFchevron_right
Using Evolutionary Computation to Facilitate Development of Neurocontrol
Tony Martinez

The field of neurocontrol, in which neural networks are used for control of complex systems, has many potential applications. One of the biggest hurdles to developing neurocontrollers is the difficulty in establishing good training data for the neural network. We propose a hybrid approach to the development of neurocontrollers that employs both evolutionary computation (EC) and neural networks (NN). The survivors of this evolutionary process are used to construct a training set for the NN. The NN learns the training set, is able to generalize to new system states, and is then used for neurocontrol. Thus the EC/NN approach combines the broad, parallel search of EC with the rapid execution and generalization of NN to produce a viable solution to the control problem. This paper presents the EC/NN hybrid and demonstrates its utility in developing a neurocontroller for the pole balancing problem.

View PDFchevron_right
Learning to do without synaptic plasticity: evolving adaptive neural networks controllers for autonomous robot
Matt Quinn, Elio Tuci

2002

Recently research in the field of Evolutionary Robotics have begun to investigate the evolution of learn- ing controllers for autonomous robots. Research in this area has achieved some promising results, but research to date has focussed on the evolution of neural networks in- corporating synaptic plasticity. There has been little inves- tigation of possible alternatives, although the importance of exploring

View PDFchevron_right
Evolution of Neural Architecture Fitting Environmental Dynamics
Genci Capi

Adaptive Behavior, 2005

Temporal and sequential information is essential to any agent continually interacting with its environment. In this paper, we test whether it is possible to evolve a recurrent neural network controller to match the dynamic requirement of the task. As a benchmark, we consider a sequential navigation task where the agent has to alternately visit two rewarding sites to obtain food and water after first visiting the nest. To achieve a better fitness, the agent must select relevant sensory inputs and update its working memory to realize a non-Markovian sequential behavior in which the preceding state alone does not determine the next action. We compare the performance of a feed-forward and recurrent neural control architectures in different environment settings and analyze the neural mechanisms and environment features exploited by the agents to achieve their goal. Simulation and experimental results using the Cyber Rodent robot show that a modular architecture with a locally excitatory recurrent layer outperformed the general recurrent controller.

View PDFchevron_right

Related topics

  • Computer Science
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved