Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Background
Conclusions
References
All Topics
Computer Science

Dual Monte Carlo Tree Search

Profile image of prashank kadamprashank kadam

2021, ArXiv

Abstract

AlphaZero, using a combination of Deep Neural Networks and Monte Carlo Tree Search (MCTS), has successfully trained reinforcement learning agents in a tabula-rasa way. The neural MCTS algorithm has been successful in finding nearoptimal strategies for games through self-play. However, the AlphaZero algorithm has a significant drawback; it takes a long time to converge and requires high computational power due to complex neural networks for solving games like Chess, Go, Shogi, etc. Owing to this, it is very difficult to pursue neural MCTS research without cutting-edge hardware, which is a roadblock for many aspiring neural MCTS researchers. In this paper, we propose a new neural MCTS algorithm, called Dual MCTS, which helps overcome these drawbacks. Dual MCTS uses two different search trees, a single deep neural network, and a new update technique for the search trees using a combination of the PUCB, a sliding-window, and the -greedy algorithm. This technique is applicable to any MCT...

Related papers

Pruning Stochastic Game Trees Using Neural Networks for Reduced Action Space Approximation
Georgios Alexandridis

Mathematics

Monte Carlo Tree Search has proved to be very efficient in the broad domain of Game AI, though it suffers from high dimensionality in cases of large branching factors. Several pruning techniques have been proposed to tackle this problem, most of which require explicit domain knowledge. In this study, an approach using neural networks to determine the number of actions to be pruned, depending on the iterations run and the total number of possible actions, is proposed. Multi-armed bandit simulations with the UCB1 formula are employed to generate suitable datasets for the networks’ training and a specifically designed process is followed to select the best combination of the number of iterations and actions for pruning. Two pruning Monte Carlo Tree Search variants are investigated, based on different actions’ expected rewards’ distributions, and they are evaluated in the collectible card game Hearthstone. The proposed technique improves the performance of the Monte Carlo Tree Search al...

View PDFchevron_right
Multi-Dimensional Deep Memory Go-Player for Parameter Exploring Policy Gradients
Tom Schaul

Developing superior artificial board-game players is a widelystudied area of Artificial Intelligence. Among the most challenging games is the Asian game of Go, which, despite its deceivingly simple rules, has eluded the development of artificial expert players. In this paper we attempt to tackle this challenge through a combination of two recent developments in Machine Learning. We employ Multi-Dimensional Recurrent Neural Networks with Long Short-Term Memory cells to handle the multi-dimensional data of the board game in a very natural way. In order to improve the convergence rate, as well as the ultimate performance, we train those networks using Policy Gradients with Parameter-based Exploration, a recently developed Reinforcement Learning algorithm which has been found to have numerous advantages over Evolution Strategies. Our empirical results confirm the promise of this approach, and we discuss how it can be scaled up to expert-level Go players.

View PDFchevron_right
Challenging Human Supremacy: Evaluating Monte Carlo Tree Search and Deep Learning for the Trick Taking Card Game Jass
Markus Stolze

2020

Despite the recent successful application of Artificial Intelligence (AI) to games, the performance of cooperative agents in imperfect information games is still far from surpassing humans. Cooperating with teammates whose play-styles are not previously known poses additional challenges to current state-of-the-art algorithms. In the Swiss card game Jass, coordination within the two opposing teams is crucial for winning. Since verbal communication is forbidden, the only way to transmit information within the team is through a player’s play-style. This makes the game a particularly suitable candidate subject to continue the research on AI in cooperation games with hidden information. In this work, we analyse the effectiveness and shortcomings of several state-of-the-art algorithms (Monte Carlo Tree Search (MCTS) variants and Deep Neural Networks (DNNs)) at playing the Jass game. Our key contributions are two-fold. First, we provide a performance overview for state-of-the-art algorithm...

View PDFchevron_right
Monte-Carlo Tree Search In Poker Using Expected Reward Distributions
Jan Ramon

Advances in Machine …, 2009

View PDFchevron_right
Learning Policies from Self-Play with Policy Gradients and MCTS Value Estimates
Cameron Browne

2019 IEEE Conference on Games (CoG)

In recent years, state-of-the-art game-playing agents often involve policies that are trained in self-playing processes where Monte Carlo tree search (MCTS) algorithms and trained policies iteratively improve each other. The strongest results have been obtained when policies are trained to mimic the search behaviour of MCTS by minimising a cross-entropy loss. Because MCTS, by design, includes an element of exploration, policies trained in this manner are also likely to exhibit a similar extent of exploration. In this paper, we are interested in learning policies for a project with future goals including the extraction of interpretable strategies, rather than state-of-the-art game-playing performance. For these goals, we argue that such an extent of exploration is undesirable, and we propose a novel objective function for training policies that are not exploratory. We derive a policy gradient expression for maximising this objective function, which can be estimated using MCTS value estimates, rather than MCTS visit counts. We empirically evaluate various properties of resulting policies, in a variety of board games.

View PDFchevron_right
Optimizations over neural MCTS for solving combinatorial problems
prashank kadam

2022

Master of Science Optimizations over Neural MCTS for solving Combinatorial problems by Prashank KADAM AlphaZero, using a combination of Deep Neural Networks and Monte Carlo Tree Search (MCTS), has successfully trained reinforcement learning agents in a tabula-rasa way. The neural MCTS algorithm has been successful in finding near-optimal strategies for games through self-play. However, the AlphaZero algorithm has a significant drawback; it takes a long time to converge due to a large number of policy updates before reaching the optimal strategy and requires high computational power due to complex neural networks for solving games like Chess, Go, Shogi, etc. Owing to this, it is very difficult to pursue neural MCTS research without cutting-edge hardware, which is a roadblock for many aspiring neural MCTS researchers. As a part of this thesis, we propose multiple techniques to accelerate the training. Along with this we also propose two new neural MCTS algorithms, called Meta MPV-MCTS and Dual MCTS, which help overcome these drawbacks. Both these algorithms use two different search trees instead of a single one in AlphaZero for the purpose of providing lookahead, along with a novel technique to reduce the number of updates made to the tree and can be extended to any MCTS based algorithms. We evaluate these enhancements along with out newly proposed algorithms over different types of symmetric as well as asymmetric games. We show various strategies through which asymmetric games can be trained more efficiently. Finally, we have developed a grammar which we call Persophone-S which can be used for defining two-player first order logic based games. This grammar can be used for defining real-world problems into games and then evaluating these definitions with our algorithms.

View PDFchevron_right
αβ-based play-outs in Monte-Carlo Tree Search
Yngvi Björnsson

2011

Abstract Monte-Carlo Tree Search (MCTS) is a recent paradigm for game-tree search, which gradually builds a game-tree in a best-first fashion based on the results of randomized simulation play-outs. The performance of such an approach is highly dependent on both the total number of simulation play-outs and their quality.

View PDFchevron_right
Learning to play Go using recursive neural networks
Lin Wu, Pierre Baldi

Go is an ancient board game that poses unique opportunities and challenges for artificial intelligence. Currently, there are no computer Go programs that can play at the level of a good human player. However, the emergence of large repositories of games is opening the door for new machine learning approaches to address this challenge. Here we develop a machine learning approach to Go, and related board games, focusing primarily on the problem of learning a good evaluation function in a scalable way. Scalability is essential at multiple levels, from the library of local tactical patterns, to the integration of patterns across the board, to the size of the board itself. The system we propose is capable of automatically learning the propensity of local patterns from a library of games. Propensity and other local tactical information are fed into recursive neural networks, derived from a probabilistic Bayesian network architecture. The recursive neural networks in turn integrate local information across the board in all four cardinal directions and produce local outputs that represent local territory ownership probabilities. The aggregation of these probabilities provides an effective strategic evaluation function that is an estimate of the expected area at the end, or at various other stages, of the game. Local area targets for training can be derived from datasets of games played by human players. In this approach, while requiring a learning time proportional to N 4 , skills learned on a board of size N 2 can easily be transferred to boards of other sizes. A system trained using only 9 × 9 amateur game data performs surprisingly well on a test set derived from 19 × 19 professional game data. Possible directions for further improvements are briefly discussed.

View PDFchevron_right
Alpha-Beta-based Play-outs in Monte-Carlo Tree Search
Yngvi Björnsson

Monte-Carlo Tree Search (MCTS) is a recent paradigm for game-tree search, which gradually builds a game-tree in a best-first fashion based on the results of randomized simulation play-outs. The performance of such an approach is highly dependent on both the total number of simulation play-outs and their quality. The two metrics are, however, typically inversely correlated — improving the quality of the play-outs generally involves adding knowledge that requires extra computation, thus allowing fewer play-outs to be performed per time unit. The general practice in MCTS seems to be more towards using relatively knowledge-light play-out strategies for the benefit of getting additional simulations done. In this paper we show, for the game Lines of Action (LOA), that this is not necessarily the best strategy. The newest version of our simulation-based LOA program, MC-LOA, uses a selective 2-ply αβ-search at each step in its play-outs for choosing a move. Even though this reduces the numb...

View PDFchevron_right
Combining Monte Carlo tree search and apprenticeship learning for capture the flag
Xiaodong Li

2015 IEEE Conference on Computational Intelligence and Games (CIG), 2015

In this paper we introduce a novel approach to agent control in competitive video games which combines Monte Carlo Tree Search (MCTS) and Apprenticeship Learning (AL). More specifically, an opponent model created through AL is used during the expansion phase of the Upper Confidence Bounds for Trees (UCT) variant of MCTS. We show how this approach can be applied to a game of Capture the Flag (CTF), an environment which is both non-deterministic and partially observable. The performance gain of a controller utilizing an opponent model learned via AL when compared to a controller using just UCT is shown both with win/loss ratios and True Skill rankings. Additionally, we build on previous findings by providing evidence of a bias towards a particular style of play in the AI Sandbox CTF environment. We believe that the approach highlighted here can be extended to a wider range of games other than just CTF.

View PDFchevron_right

References (17)

  1. Elo, A. E. The rating of chessplayers, past and present. 1978.
  2. Frankl, P. and Tokushige, N. The game of n-times nim. Discrete Math., 260(1-3):205-209, January 2003. ISSN 0012-365X. doi: 10.1016/S0012-365X(02)00667-2.
  3. Gelly and Silver, D. Monte carlo tree search and rapid action value estimation in computer go. Artificial Intelligence, 175, 2011.
  4. Grill, J.-B., Altché, F., Tang, Y., Hubert, T., Valko, M., Antonoglou, I., and Munos, R. Monte-carlo tree search as regularized policy optimization. ArXiv, abs/2007.12509, 2020.
  5. Hill, G. and Kemp, S. M. Connect 4: A novel paradigm to elicit positive and negative insight and search problem solving. 9(1755), 2018.
  6. Hodges, W. and Väänänen, J. Logic and Games. In Zalta, E. N. (ed.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, fall 2019 edition, 2019.
  7. Kleinberg, J. and Tardos, E. Algorithm Design. Addison- Wesley Longman Publishing Co., Inc., USA, 2005. ISBN 0321295358.
  8. Lan, L.-C., Li, W., Wei, T.-H., and Wu, I. Multiple policy value monte carlo tree search. In IJCAI, 2019.
  9. Lanctot, M., Winands, M. H., Pepels, T., and Sturtevant, N. R. Monte carlo tree search with heuristic evaluations using implicit minimax backups. IEEE Conference on Computational Intelligence and Games, pp. 1-8, 2014.
  10. Lanctot, M., Lockhart, E., Lespiau, J.-B., Zambaldi, V., Upadhyay, S., Pérolat, J., Srinivasan, S., Timbers, F., Tuyls, K., Omidshafiei, S., Hennes, D., Morrill, D., Muller, P., Ewalds, T., Faulkner, R., Kramár, J., Vylder, B. D., Saeta, B., Bradbury, J., Ding, D., Borgeaud, S., Lai, M., Schrittwieser, J., Anthony, T., Hughes, E., Dani- helka, I., and Ryan-Davis, J. Openspiel: A framework for reinforcement learning in games. 2019.
  11. Omidshafiei, S., Papadimitriou, C., Piliouras, G., Tuyls, K., Rowland, M., Lespiau, J.-B., Czarnecki, W. M., Lanc- tot, M., Perolat, J., and Munos, R. α-rank: Multi-agent evaluation by evolution. Nature, 1038, Jul 2019.
  12. Rosin, C. D. Multi-armed bandits with episode context. volume 61, pp. 203-230, 2011.
  13. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Driessche, G. V. D., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., and Lanctot, M. Mastering the game of go with deep neural networks and tree search. Nature, 529, Jul 2016a.
  14. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D. Mastering the game of go with deep neural networks and tree search. Nature, 529:484,503, 2016b.
  15. Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., and Hassabis, D. Mastering the game of go without human knowledge. Nature, 550:354, Oct 2017.
  16. Xu, R. and Lieberherr, K. Learning self-game-play agents for combinatorial optimization problems. AAMAS, Pro- ceedings of the 18th International Conference on Au- tonomous Agents and MultiAgent Systems, pp. 2276- 2278, 2019.
  17. Xu, R. and Lieberherr, K. Learning self-play agents for combinatorial optimization problems. The Knowl- edge Engineering Review, 35:e11, 2020. doi: 10.1017/ S026988892000020X.

Related papers

AlphaZero-Inspired Game Learning: Faster Training by Using MCTS Only at Test Time
Wolfgang Konen

IEEE transactions on games, 2022

Recently, the seminal algorithms AlphaGo and Al-phaZero have started a new era in game learning and deep reinforcement learning. While the achievements of AlphaGo and AlphaZero-playing Go and other complex games at super human level-are truly impressive, these architectures have the drawback that they require high computational resources. Many researchers are looking for methods that are similar to AlphaZero, but have lower computational demands and are thus more easily reproducible. In this paper, we pick an important element of AlphaZerothe Monte Carlo Tree Search (MCTS) planning stage-and combine it with temporal difference (TD) learning agents. We wrap MCTS for the first time around TD n-tuple networks and we use this wrapping only at test time to create versatile agents that keep at the same time the computational demands low. We apply this new architecture to several complex games (Othello, ConnectFour, Rubik's Cube) and show the advantages achieved with this AlphaZero-inspired MCTS wrapper. In particular, we present results that this agent is the first one trained on standard hardware (no GPU or TPU) to beat the very strong Othello program Edax up to and including level 7 (where most other learning-from-scratch algorithms could only defeat Edax up to level 2).

View PDFchevron_right
Mastering the game of Go with deep neural networks and tree search
Tarun Gogineni

All games of perfect information have an optimal value function, v * (s), which determines the outcome of the game, from every board position or state s, under perfect play by all players. These games may be solved by recursively computing the optimal value function in a search tree containing approximately b d possible sequences of moves, where b is the game's breadth (number of legal moves per position) and d is its depth (game length). In large games, such as chess (b ≈ 35, d ≈ 80) 1 and especially Go (b ≈ 250, d ≈ 150) 1 , exhaustive search is infeasible 2,3 , but the effective search space can be reduced by two general principles. First, the depth of the search may be reduced by position evaluation: truncating the search tree at state s and replacing the subtree below s by an approximate value function v(s) ≈ v * (s) that predicts the outcome from state s. This approach has led to superhuman performance in chess 4 , checkers 5 and othello 6 , but it was believed to be intractable in Go due to the complexity of the game 7. Second, the breadth of the search may be reduced by sampling actions from a policy p(a|s) that is a probability distribution over possible moves a in position s. For example, Monte Carlo rollouts 8 search to maximum depth without branching at all, by sampling long sequences of actions for both players from a policy p. Averaging over such rollouts can provide an effective position evaluation, achieving superhuman performance in backgammon 8 and Scrabble 9 , and weak amateur level play in Go 10. Monte Carlo tree search (MCTS) 11,12 uses Monte Carlo rollouts to estimate the value of each state in a search tree. As more simulations are executed, the search tree grows larger and the relevant values become more accurate. The policy used to select actions during search is also improved over time, by selecting children with higher values. Asymptotically, this policy converges to optimal play, and the evaluations converge to the optimal value function 12. The strongest current Go programs are based on MCTS, enhanced by policies that are trained to predict human expert moves 13. These policies are used to narrow the search to a beam of high-probability actions, and to sample actions during rollouts. This approach has achieved strong amateur play 13–15. However, prior work has been limited to shallow policies 13–15 or value functions 16 based on a linear combination of input features. Recently, deep convolutional neural networks have achieved unprecedented performance in visual domains: for example, image classification 17 , face recognition 18 , and playing Atari games 19. They use many layers of neurons, each arranged in overlapping tiles, to construct increasingly abstract, localized representations of an image 20. We employ a similar architecture for the game of Go. We pass in the board position as a 19 × 19 image and use convolutional layers to construct a representation of the position. We use these neural networks to reduce the effective depth and breadth of the search tree: evaluating positions using a value network, and sampling actions using a policy network. We train the neural networks using a pipeline consisting of several stages of machine learning (Fig. 1). We begin by training a supervised learning (SL) policy network p σ directly from expert human moves. This provides fast, efficient learning updates with immediate feedback and high-quality gradients. Similar to prior work 13,15 , we also train a fast policy p π that can rapidly sample actions during rollouts. Next, we train a reinforcement learning (RL) policy network p ρ that improves the SL policy network by optimizing the final outcome of games of self-play. This adjusts the policy towards the correct goal of winning games, rather than maximizing predictive accuracy. Finally, we train a value network v θ that predicts the winner of games played by the RL policy network against itself. Our program AlphaGo efficiently combines the policy and value networks with MCTS. Supervised learning of policy networks For the first stage of the training pipeline, we build on prior work on predicting expert moves in the game of Go using supervised learning 13,21–24. The SL policy network p σ (a | s) alternates between con-volutional layers with weights σ, and rectifier nonlinearities. A final soft-max layer outputs a probability distribution over all legal moves a. The input s to the policy network is a simple representation of the board state (see Extended Data Table 2). The policy network is trained on randomly The game of Go has long been viewed as the most challenging of classic games for artificial intelligence owing to its enormous search space and the difficulty of evaluating board positions and moves. Here we introduce a new approach to computer Go that uses 'value networks' to evaluate board positions and 'policy networks' to select moves. These deep neural networks are trained by a novel combination of supervised learning from human expert games, and reinforcement learning from games of self-play. Without any lookahead search, the neural networks play Go at the level of state-of-the-art Monte Carlo tree search programs that simulate thousands of random games of self-play. We also introduce a new search algorithm that combines Monte Carlo simulation with value and policy networks. Using this search algorithm, our program AlphaGo achieved a 99.8% winning rate against other Go programs, and defeated the human European Go champion by 5 games to 0. This is the first time that a computer program has defeated a human professional player in the full-sized game of Go, a feat previously thought to be at least a decade away.

View PDFchevron_right
Improving Monte Carlo Tree Search with Artificial Neural Networks without Heuristics
Cristian González García

2021

Monte Carlo Tree Search is one of the main search methods studied presently. It has demonstrated its efficiency in the resolution of many games such as Go or Settlers of Catan and other different problems. There are several optimizations of Monte Carlo, but most of them need heuristics or some domain language at some point, making very difficult its application to other problems. We propose a general and optimized implementation of Monte Carlo Tree Search using neural networks without extra knowledge of the problem. As an example of our proposal, we made use of the Dots and Boxes game. We tested it against other Monte Carlo system which implements specific knowledge for this problem. Our approach improves accuracy, reaching a winning rate of 81% over previous research but the generalization penalizes performance.

View PDFchevron_right
Monte Carlo Tree Search for Policy Optimization
Xiaobai Ma

2019

Gradient-based methods are often used for policy optimization in deep reinforcement learning, despite being vulnerable to local optima and saddle points. Although gradient-free methods (e.g., genetic algorithms or evolution strategies) help mitigate these issues, poor initialization and local optima are still concerns in highly nonconvex spaces. This paper presents a method for policy optimization based on Monte-Carlo tree search and gradient-free optimization. Our method, called Monte-Carlo tree search for policy optimization (MCTSPO), provides a better exploration-exploitation trade-off through the use of the upper confidence bound heuristic. We demonstrate improved performance on reinforcement learning tasks with deceptive or sparse reward functions compared to popular gradient-based and deep genetic algorithm baselines.

View PDFchevron_right
Deep reinforcement learning compared with Q-table learning applied to backgammon
Max Winberg

2016

Reinforcement learning attempts to mimic how humans react to their surrounding environment by giving feedback to software agents based on the actions they take. To test the capabilities of these agents, researches have long regarded board games as a powerful tool. This thesis compares two approaches to reinforcement learning in the board game backgammon, a Q-table and a deep reinforcement network. It was determined which approach surpassed the other in terms of accuracy and convergence rate towards the perceived optimal strategy. The evaluation is performed by training the agents using the self-learning approach. After variable amounts of training sessions, the agents are benchmarked against each other and a third, random agent. The results derived from the study indicate that the convergence rate of the deep learning agent is far superior to that of the Q-table agent. However, the results also indicate that the accuracy of Q-tables is greater than that of deep learning once the for...

View PDFchevron_right

Related topics

  • Computer Science
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved