Human and Computer Preferences at Chess

Advances in Computer Games, 2010

Evaluating agents in decision-making applications requires assessing their skill and predicting their behaviour. Both are well developed in Poker-like situations, but less so in more complex game and model domains. This paper addresses both tasks by using Bayesian inference in a benchmark space of reference agents. The concepts are explained and demonstrated using the game of chess but the model applies generically to any domain with quantifiable options and fallible choice. Demonstration applications address questions frequently asked by the chess community regarding the stability of the rating scale, the comparison of players of different eras and/or leagues, and controversial incidents possibly involving fraud. The last include alleged under-performance, fabrication of tournament results, and clandestine use of computer advice during competition. Beyond the model world of games, the aim is to improve fallible human performance in complex, high-value tasks.

IEEE Access, 2018

Since 2010, annual StarCraft artificial intelligence (AI) competitions have promoted the development of successful AI players for complex real-time strategy games. In these competitions, AI players are ranked based on their win ratio over thousands of head-to-head matches. Although simple and easily implemented, this evaluation scheme may less adequately help develop more human-competitive AI players. In this paper, we recruited 45 human StarCraft players at different expertise levels (expert/medium/novice) and asked them to play against the 18 top AI players selected from the five years of competitions (2011-2015). The results show that the human evaluations of AI players differ substantially from the current standard evaluation and ranking method. In fact, from a human standpoint, there has been little progress in the quality of StarCraft AI players over the years. It is even possible that AI-only tournaments can lead to AIs being created that are unacceptable competitors for humans. This paper is the first to systematically explore the human evaluation of AI players, the evolution of AI players, and the differences between human perception and tournament-based evaluations. The discoveries from this paper can support AI developers in game companies and AI tournament organizers to better incorporate the perspective of human users into their AI systems. INDEX TERMS Video game, Starcraft, game, artificial intelligence, game AI competition, human factor, human computer interaction.

1999

Current-generation online games typically incorporate a "computer" opponent t o train n ew players to compete against human opponents. The quality of this training depends to a large degree on how similar the computer's play is to that of an experienced hu man player. For instance, inhuman weaknesses in computer play encourage new players to develop tactics, prediction rules and

2013 IEEE Conference on Computational Inteligence in Games (CIG), 2013

We build a model for the kind of decision making involved in games of strategy such as chess, making it abstract enough to remove essentially all game-specific contingency, and compare it to known psychometric models of test taking, item response, and performance assessment. Decisions are modeled in terms of fallible agents Z faced with possible actions ai whose utilities ui = u(ai) are not fully apparent. The three main goals of the model are prediction, meaning to infer probabilities pi for Z to choose ai; intrinsic rating, meaning to assess the skill of a person's actual choices ai t over various test items t; and simulation of the distribution of choices by an agent with a specified skill set. We describe and train the model on large data from chess tournament games of different ranks of players, and exemplify its accuracy by applying it to give intrinsic ratings for world championship matches.

Proceedings of AAAI, 2011

This paper develops and tests formulas for representing playing strength at chess by the quality of moves played, rather than the results of games. Intrinsic quality is estimated via evaluations given by computer chess programs run to high depth, ideally whose playing strength is sufficiently far ahead of the best human players as to be a “relatively omniscient” guide. Several formulas, each having intrinsic skill parameters s for “sensitivity” and c for “competence,” are argued theoretically and tested by regression on large sets of tournament games played by humans of varying strength as measured by the internationally standard Elo rating system. This establishes a correspondence between Elo rating and the parameters. A smooth correspondence is shown between statistical results and the century points on the Elo scale, and ratings are shown to have stayed quite constant over time (i.e., little or no “inflation”). By modeling human players of various strengths, the model also enables distributional prediction to detect cheating by getting computer advice during games. The theory and empirical results are in principle transferable to other rational-choice settings in which the alternatives have well-defined utilities, but bounded information and complexity constrain the perception of the utilitiy values.

Revista Brasileira de Computação Aplicada

Computational modeling has enabled researchers to simulate tasks which are very often impossible in practice, such as deciphering the working of the human mind, and chess is used by many cognitive scientists as an investigative tool in studies on intelligence, behavioral patterns and cognitive development and rehabilitation. Computer analysis of databases with millions of chess games allows players’ cognitive development to be predicted and their behavioral patterns to be investigated. However, computers are not yet able to solve chess problems in which human intelligence analyzes and evaluates abstractly without the need for many concrete calculations. The aim of this article is to describe and simulate a chess problem situation proposed by the British mathematician Sir Roger Penrose and thus provide an opportunity for a comparative discussion by society of human and artificial intelligence. To this end, a specialist chess computer program, Fritz 12, was used to simulate possible m...

Behavioral Science, 1980

In this paper we use the analytic hierarchy process to combine technical and behavioral characteristics of chess players and predict the outcome of a championship match. The method also applies to decision making in living systems at the level of the group. Our approach to prediction deals with the inputs (know-how of the players) of a system (the chess board and its psychological surrounding environment), the transformation of the input within the system through actual play, and with its outputs (outcome of the match). Thus our task is to assess the quality of the input by deriving a relative index of power of the players by identifying (with the assistance of grandmasters through a questionnaire) the relevant factors whose totality may determine the outcome. We then use this index to assess the kind of output (win, draw, or loss) that it would produce over a set of several encounters in a match. The outcome of our analysis is an estimate of the total number of games that two given players would play, together with the numbers of games won by each player. We also analyze the sensitivity of the results to changes in the expectations of the competitors.

cse.buffalo.edu

This paper presents evidence for several features of the population of chess players, and the distribution of their performances measured in terms of Elo ratings and by computer analysis of moves. Evidence that ratings have remained stable since the inception of the Elo system in the 1970’s is given in several forms: by showing that the population of strong players fits a simple logistic-curve model without inflation, by plotting players’ average error against the FIDE category of tournaments over time, and by skill parameters from a model that employs computer analysis keeping a nearly constant relation to Elo rating across that time. The distribution of the model’s Intrinsic Performance Ratings can hence be used to compare populations that have limited interaction, such as between players in a national chess federation and FIDE, and ascertain relative drift in their respective rating systems.

Arxiv preprint cs/0611163, 2006

Abstract: We investigate systematically the impact of human intervention in the training of computer players in a strategy board game. In that game, computer players utilise reinforcement learning with neural networks for evolving their playing strategies and demonstrate a slow learning speed. Human intervention can significantly enhance learning performance, but carry-ing it out systematically seems to be more of a problem of an integrated game development environment as opposed to automatic evolutionary learning.

2005

Anecdotal evidence exists that in many positions two distinct chess engines will choose different moves and, moreover, that their top-n ranking of move choices also differ. Here we set out to quantify this difference, including the difference between move choices by chess engines and those made by humans. For our analysis we used FRITZ 8 and JUNIOR 9 as representative chess search engines and the POWERBOOK opening book as representing human choices. We collected the top-5 ranked moves and their scores as reported by FRITZ and JUNIOR, after 15 and 30 minutes of thinking time, and the top-5 moves recorded in the POWERBOOK, for the Nunn2 test positions and the initial board position. The data analysis was carried out using several nonparametric measures, including the amount of overlap in the top-5 choices of the engines and their association as measured by three variants of Spearman's footrule. Our preliminary results show that, overall, the engines differ substantially in their choice of moves, and, furthermore, the engines' choices also differ substantially from human choice.