Learning in Robot Football

Lecture Notes in Computer Science, 2015

This paper shows the results of applying machine learning techniques to the problem of predicting soccer plays in the Small Size League of RoboCup. We have modeled the task as a multi-class classification problem by learning the plays of the STOx's team. For this, we have created a database of observations for this team's plays and obtained key features that describe the game state during a match. We have shown experimentally, that these features allow two learning classifiers to obtain high prediction accuracies and that most miss-classified observations are found early on the plays.

A central problem in autonomous robotics is how to design programs that determine what the robot should do next. Behaviour-based control is a popular paradigm, but current approaches to behaviour design typically involve hand-coded behaviours. The aim of this work is to explore the use of reinforcement learning to develop autonomous robot behaviours automatically, and specifically to look at the performance of the resulting behaviours. This thesis examines the question of whether behaviours for a real behaviour-based, autonomous robot can be learnt under simulation using the Monte Carlo Exploring Starts, -soft On Policy Monte Carlo or linear, gradient-descent Sarsa algorithms. A further question is whether the increased performance of learnt behaviours carries through to increased performance on the real robot. In addition, this work looks at whether continuing to learn on the real robot causes further improvement in the performance of the behaviour. A novel method is developed, termed Policy Initialisation, that makes use of the domain knowledge in an existing, hand-coded behaviour by converting the behaviour into either a reinforcement learning policy or an action-value function. This is then used to bootstrap the learning process. The Markov Decision Process model is central to reinforcement learning algorithms. This work examines whether it is possible to use an internal world model in the real robot to suit the requirements of the Markov Decision Process model. The methodology used to answer these questions is to take three realistic, non-trivial robotic tasks, and attempt to learn behaviours for each. The learnt behaviours are then compared with hand-coded behaviours that have either been published or used in international competition. The tasks are based on real task requirements for robots used in a RoboCup Formula 2000 robot soccer team. The first is a generic movement behaviour that moves the robot to a target point. The second requires the robot to dribble the ball in an arc so that the robot maintains possession and so that the final position is lined up with the goal. The third addresses the problem of kicking the ball away from the wall. The results show that for these three different types of behavioural problem, reinforcement learning on a simulator produced significantly better performance than hand-coded equivalents, not only under simulation but also on the real robot. In contrast to this, continuing the learning process on the real robot did not significantly improve performance. The Policy Initialisation technique is found to accelerate learning for tabular Monte Carlo methods, but makes minimal improvement and is, in fact, costly to use in conjunction with linear, gradient-descent Sarsa. This approach, unlike some other techniques for accelerating learning, does not appear to bias the solution. Finally, the evidence from this thesis is that internal world models that maintain the requirements of Markov Decision Processes can be constructed, and this appears to be a sound approach to avoiding problems connected with partial observability that have previously occurred in the use of reinforcement learning in robotic environments.

The Robocup 97 competition provides an opportunity to demonstrate the techniques and methods of artificial intelligence, autonomous agents and computer vision. In this competition each of the robots (or agents) must know how to dribble, shoot, pass, and recover the ball from an opponent. Each agent must also be able to evaluate its position with respect to its teammates and opponents, and then decide whether to wait for a pass, run for the ball, cover an opponent's attack, or go to help a teammate; while at the same time following the rules of the game. The most important feature of the USC/ISI robot soccer team is that every robot is autonomous and self-contained with all of its essential capabilities on-board. There are three types of soccer playing position that a robot can be: goal keeper, defender and forward.

… Robots, 2006 6th …, 2006

Robotic soccer superseded chess as a challenge problem and benchmark for artificial intelligence research and poses many challenges for robotics. The international RoboCup championships grew to the most important robotic competition worldwide. This paper describes the mechanical and electrical design of the robots that we constructed for RoboCup 2006, which took place in Bremen, Germany. The paper also covers the software used for perception, behavior control, communication, and simulation. Our robots performed well. The KidSize robots won the Penalty Kick competition and came in second the overall Best Humanoid ranking, next only to the titleholder, Team Osaka.

1995

Abstract The range of unseen instances that can be successfully classi ed by a learning algorithm is determined not only by the distribution of the training data, but also by the parameters of the function to be learned. With the right parameters, learning in a certain region of the state space can generalize to completely di erent areas with no retraining. We demonstrate the power of using well-chosen inputs to (and outputs from) neural networks by conducting experiments in our robotic soccer domain.

Lecture Notes in Computer Science, 1998

The Robocup 97 competition provides an excellent opportunity to demonstrate the techniques and methods of artificial intelligence, autonomous agents and computer vision. On a soccer field the core capabilities a player must have are to navigate the field, track the ball and other agents, recognize the difference between agents, collaborate with other agents, and hit the ball in the correct direction. USC's Dreamteam of robots can be described as a group of mobile autonomous agents collaborating in a rapidly changing environment. The key characteristic of this team is that each soccer robot is an autonomous agent, self-contained with all of its essential capabilities on-board. Our robots share the same general architecture and basic hardware, but they have integrated abilities to play different roles (goalkeeper, defender or forward) and utilize different strategies in their behavior. Our philosophy in building these robots is to use the least possible sophistication to make them as robust as possible. In the 1997 RoboCup competition, the Dreamteam played well and won the world championship in the middlesized robot league.

Lecture Notes in Computer Science, 2010

Robotics and Autonomous Systems, 2009

Humanoid robots are enjoying increasing popularity as a research tool. As step towards the long-term goal of winning against the FIFA world champion, the RoboCup Federation added in 2002 a league for humanoid robots to its annual soccer competitions. In this paper, the Humanoid League competitions that took place so far are reviewed. The different approaches for the design of robot hardware and the software for perception and behavior control of soccer playing humanoid robots are discussed. The paper concludes with an outlook to the future of the RoboCup Humanoid League.

International Journal of Humanoid Robotics, 2008

Humanoid robots are enjoying increasing popularity as a research tool. As step towards the long-term goal of winning against the FIFA world champion, the RoboCup Federation added in 2002 a league for humanoid robots to its annual soccer competitions. In this paper, the Humanoid League competitions that took place so far are reviewed. The different approaches for the design of robot hardware and the software for perception and behavior control of soccer playing humanoid robots are discussed. The paper concludes with an outlook to the future of the RoboCup Humanoid League.

AI magazine, 2001