Psychometric modeling of decision making via game play

We investigate the question of experts’ ability to estimate task difficulty through a case study that asks players to rate tactical chess positions. In an eye tracking experiment, experts’ estimations are compared to the statistic–based difficulty ratings of the chesstempo.com website. The subjects’ solutions of chess problems and their considered chess variations are analyzed in connection to ChessTempo’s solutions. In addition, eye tracking and performance data (time and accuracy) are used as physiological indicators of subjectively perceived difficulty. In the course of our research, we also aim to identify the attributes of tactical positions that induce difficulty. Understanding the connection between players’ estimation of difficulty and the properties of the search trees of variations considered is essential for modeling the difficulty of tactical positions.

… Intelligence and Data …, 2009

Systems Engineering often involves computer modelling the behaviour of proposed systems and their components. Where a component is human, fallibility must be modelled by a stochastic agent. The identification of a model of decision-making over quantifiable options is investigated using the game-domain of Chess. Bayesian methods are used to infer the distribution of players’ skill levels from the moves they play rather than from their competitive results. The approach is used on large sets of games by players across a broad FIDE Elo range, and is in principle applicable to any scenario where high-value decisions are being made under pressure.

2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), 2015

Inferences about structured patterns in human decision making have been drawn from medium-scale simulated competitions with human subjects. The concepts analyzed in these studies include level-k thinking, satisficing, and other human error tendencies. These concepts can be mapped via a natural depth of search metric into the domain of chess, where copious data is available from hundreds of thousands of games by players of a wide range of precisely known skill levels in real competitions. The games are analyzed by strong chess programs to produce authoritative utility values for move decision options by progressive deepening of search. Our experiments show a significant relationship between the formulations of level-k thinking and the skill level of players. Notably, the players are distinguished solely on moves where they erred-according to the average depth level at which their errors are exposed by the authoritative analysis. Our results also indicate that the decisions are often independent of tail assumptions on higher-order beliefs. Further, we observe changes in this relationship in different contexts, such as minimal versus acute time pressure. We try to relate satisficing to insufficient level of reasoning and answer numerically the question, why do humans blunder?

IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2013

This paper proposes and demonstrates an approach, Skilloscopy, to the assessment of decision makers. In an increasingly sophisticated, connected and information-rich world, decision making is becoming both more important and more difficult. At the same time, modelling decision-making on computers is becoming more feasible and of interest, partly because the information-input to those decisions is increasingly on record. The aims of Skilloscopy are to rate and rank decision makers in a domain relative to each other: the aims do not include an analysis of why a decision is wrong or suboptimal, nor the modelling of the underlying cognitive process of making the decisions.

2007 IEEE Symposium on Computational Intelligence and Games, 2007

Accuracy of evaluation functions is one of the critical factors in computer game players. Evaluation functions are usually constructed manually as a weighted linear combination of evaluation features that characterize game positions. Selecting evaluation features and tuning their weights require deep knowledge of the game and largely alleviates such efforts.

Proceedings of the International Conference on Agents and Artificial Intelligence, 2015

Research on judging decisions made by fallible (human) agents is not as much advanced as research on finding optimal decisions. Human decisions are often influenced by various factors, such as risk, uncertainty, time pressure, and depth of cognitive capability, whereas decisions by an intelligent agent (IA) can be effectively optimal without these limitations. The concept of 'depth', a well-defined term in game theory (including chess), does not have a clear formulation in decision theory. To quantify 'depth' in decision theory, we can configure an IA of supreme competence to 'think' at depths beyond the capability of any human, and in the process collect evaluations of decisions at various depths. One research goal is to create an intrinsic measure of the depth of thinking required to answer certain test questions, toward a reliable means of assessing their difficulty apart from item-response statistics. We relate the depth of cognition by humans to depths of search, and use this information to infer the quality of decisions made, so as to judge the decision-maker from his decisions. We use large data from real chess tournaments and evaluations from chess programs (AI agents) of strength beyond all human players. We then seek to transfer the results to other decision-making fields in which effectively optimal judgments can be obtained from either hindsight, answer banks, powerful AI agents or from answers provided by judges of various competency.

Proceedings of the 7th International Conference on Human-Agent Interaction

Human chess players prefer training with human opponents over chess agents as the latter are distinctively different in level and style than humans. Chess agents designed for human-agent play are capable of adjusting their level, however their style is not aligned with that of human players. In this paper, we propose a novel approach for designing such agents by integrating the theory of chess players' decision-making with a state-of-the-art Monte Carlo Tree Search (MCTS) algorithm. We demonstrate the benefits of our approach using two sets of analyses. Quantitatively, we establish that the agents attain their desired Elo ratings. Qualitatively, through a Turing-inspired test with a human chess expert, we show that our agents are indistinguishable from human players. CCS CONCEPTS • Computing methodologies → Artificial intelligence; • Applied computing → Computer games.

2023

This paper reviews the many cognitive processes involved with the game of chess. These processes vary from novice players to grandmasters. Chess is often used as a marker for intelligence in humans and thus it is a baseline for AI research and the use of ML models (Ensmenger, 2011). This paper focuses on two key aspects of chess: the cognitive processes involved in gaining chess expertise and the use of computers in the chess world. Analysis of cognitive processes in chess includes topics such as pattern recognition, chunking in memory, and problem-solving. The paper will further analyze how machines learn to play chess and the differences between machine and human players. With proper understanding of chess and the cognitive processes involved with playing, chess can be used as a tool to facilitate the gaining of expertise in other domains.

IEEE Transactions on Computational Intelligence and AI in Games, 2000

The Elo system for rating chess players, also used in other games and sports, was adopted by the World Chess Federation over four decades ago. Although not without controversy, it is accepted as generally reliable and provides a method for assessing players' strengths and ranking them in official tournaments.

Lecture Notes in Computer Science, 2012

This paper develops and tests formulas for representing playing strength at chess by the quality of moves played, rather than by the results of games. Intrinsic quality is estimated via evaluations given by computer chess programs run to high depth, ideally so that their playing strength is sufficiently far ahead of the best human players as to be a 'relatively omniscient' guide. Several formulas, each having intrinsic skill parameters s for sensitivity and c for consistency, are argued theoretically and tested by regression on large sets of tournament games played by humans of varying strength as measured by the internationally standard Elo rating system. This establishes a correspondence between Elo rating and the parameters. A smooth correspondence is shown between statistical results and the century points on the Elo scale, and ratings are shown to have stayed quite constant over time. That is, there has been little or no 'rating inflation'. The theory and empirical results are transferable to other rationalchoice settings in which the alternatives have well-defined utilities, but in which complexity and bounded information constrain the perception of the utility values.