Exploiting structure in policy construction

2012

Abstract: In this paper, we consider planning in stochastic shortest path (SSP) problems, a subclass of Markov Decision Problems (MDP). We focus on medium-size problems whose state space can be fully enumerated. This problem has numerous important applications, such as navigation and planning under uncertainty. We propose a new approach for constructing a multi-level hierarchy of progressively simpler abstractions of the original problem.

Proceedings of the Fifteenth …, 1999

Recently, structured methods for solving factored Markov decisions processes (MDPs) with large state spaces have been proposed recently to allow dynamic programming to be applied without the need for complete state enumeration. We propose and examine a new value iteration algorithm for MDPs that uses algebraic decision diagrams (ADDs) to represent value functions and policies, assuming an ADD input representation of the MDP. Dynamic programming is implemented via ADD manipulation. We demonstrate our method on a class of large MDPs (up to 63 million states) and show that significant gains can be had when compared to tree-structured representations (with up to a thirty-fold reduction in the number of nodes required to represent optimal value functions).

2008 Seventh Mexican International Conference on Artificial Intelligence, 2008

In this paper we present a new approach for the solution of Markov decision processes based on the use of an abstraction technique over the action space, which results in a set of abstract actions. Markovian processes have successfully solved many probabilistic problems such as: process control, decision analysis and economy. But for problems with continuous or high dimensionality domains, high computational complexity arises because the search space grows exponentially with the number of variables. In order to reduce computational complexity, our approach avoids the use of the whole domain actions during value iteration, calculating instead over the abstract actions that really operate on each state, as a state function. Our experimental results on a robot path planning task show an important reduction of computational complexity.

Proceedings of the Tenth Conference on …, 1994

We investigate planning in time-critical domains represented as Markov Decision Processes. To reduce the computational cost of the algorithm we execute actions as we construct the plan, and sacrifice optimality by searching to a fixed depth and using a heuristic function to estimate the value of states. Although this paper concentrates on the search procedure, we also discuss ways of constructing heuristic functions that are suitable for this approach.

The problem of planning under uncertainty is one of the most important elements of a successful operation. In this context, planning that only accounts for a static, preconceived adversary will not suffice. Instead, an analysis of evolving enemy's centers of gravity and the available means of attacking those centers is necessary. This latter approach provides better estimates of the enemy's plans and strategic operations; it captures an evolving picture of the interactive and inter-related dynamics of the organization and the adversary.

2021 IEEE International Conference on Robotics and Automation (ICRA)

Autonomous systems often use approximate planners that exploit state abstractions to solve large MDPs in real-time decision-making problems. However, these planners can eliminate details needed to produce effective behavior in autonomous systems. We therefore propose a novel model, a partially abstract MDP, with a set of abstract states that each compress a set of ground states to condense irrelevant details and a set of ground states that expand from a set of expanded abstract states to retain relevant details. This paper offers (1) a definition of a partially abstract MDP that (2) generalizes its ground MDP and its abstract MDP and exhibits bounded optimality depending on its abstract MDP along with (3) a lazy algorithm for planning and execution in autonomous systems. The result is a scalable approach that computes near-optimal solutions to large problems in minutes rather than hours.

We discuss NMRDPP, a system for solving decision processes with non-Markovian reward. More specifically, target decision processes exhibit Markovian dynamics and rewarding behaviours are modelled as state trajectories specified in a linear temporal logic. In addition to implementing structured, tabular and online MDP solution algorithms, NMRDPP can exploit domain specific control knowledge. State trajectories which violate the users knowledge/intuition regarding useful dynamics can be pruned from consideration by the MDP solution algorithm. Thus, in addition to facilitating concise specification of complex reward structures, NMRDPP can be used to greatly speed up policy computation for propositional MDPs. To our knowledge, NMRDPP is the only implementation of solution algorithms designed to solve decision processes with non-Markovian rewards.

Proceedings of the 23rd international conference on Machine learning - ICML '06, 2006

Recent decision-theoric planning algorithms are able to find optimal solutions in large problems, using Factored Markov Decision Processes (fmdps). However, these algorithms need a perfect knowledge of the structure of the problem. In this paper, we propose sdyna, a general framework for addressing large reinforcement learning problems by trial-and-error and with no initial knowledge of their structure. sdyna integrates incremental planning algorithms based on fmdps with supervised learning techniques building structured representations of the problem. We describe spiti, an instantiation of sdyna, that uses incremental decision tree induction to learn the structure of a problem combined with an incremental version of the Structured Value Iteration algorithm. We show that spiti can build a factored representation of a reinforcement learning problem and may improve the policy faster than tabular reinforcement learning algorithms by exploiting the generalization property of decision tree induction algorithms.

2007

Markov decision processes (MDPs) are a general framework used in artificial intelligence (AI) to model decision theoretic planning problems. Solving real world MDPs has been a major and challenging research topic in the AI literature, since classical dynamic programming algorithms converge slowly. We discuss two approaches in expediting dynamic programming. The first approach combines heuristic search strategies with dynamic programming to expedite the convergence process. The second makes use of graphical structures in MDPs to perform dynamic programming in a better order.

Planning problems where effects of actions are non-deterministic can be modeled as Markov decision processes. Planning problems are usually goal-directed. This paper proposes several techniques for exploiting the goal-directedness to accelerate value iteration, a standard algorithm for solving Markov decision processes. Empirical studies have shown that the techniques can bring about significant speedups.