Identification of Optimal Policies in Markov Decision Processes

2020

Policy gradient methods are among the most effective methods in challenging reinforcement learning problems with large state and/or action spaces. However, little is known about even their most basic theoretical convergence properties, including: if and how fast they converge to a globally optimal solution (say with a sufficiently rich policy class); how they cope with approximation error due to using a restricted class of parametric policies; or their finite sample behavior. Such characterizations are important not only to compare these methods to their approximate value function counterparts (where such issues are relatively well understood, at least in the worst case), but also to help with more principled approaches to algorithm design. This work provides provable characterizations of computational, approximation, and sample size issues with regards to policy gradient methods in the context of discounted Markov Decision Processes (MDPs). We focus on both: 1) "tabular" ...

Planning problems where effects of actions are non-deterministic can be modeled as Markov decision processes. Planning problems are usually goal-directed. This paper proposes several techniques for exploiting the goal-directedness to accelerate value iteration, a standard algorithm for solving Markov decision processes. Empirical studies have shown that the techniques can bring about significant speedups.

CONIELECOMP 2012, 22nd International Conference on Electrical Communications and Computers, 2012

In this paper we present a new approach for the solution of Markov decision processes based on the use of an abstraction technique over the action space, which results in a set of abstract actions. Markovian processes have successfully solved many probabilistic problems such as: process control, decision analysis and economy. But for problems with continuous or high dimensionality domains, high computational complexity arises because the search space grows exponentially with the number of variables. In order to reduce computational complexity, our approach avoids the use of the whole domain actions during value iteration, calculating instead over the abstract actions that really operate on each state, as a state function. Our experimental results on a robot path planning task show an important reduction of computational complexity.

Automatica, 2010

We generalize and build on the PAC Learning framework for Markov Decision Processes developed in Jain and Varaiya (2006). We consider the reward function to depend on both the state and the action. Both the state and action spaces can potentially be countably infinite. We obtain an estimate for the value function of a Markov decision process, which assigns to each policy its expected discounted reward. This expected reward can be estimated as the empirical average of the reward over many independent simulation runs. We derive bounds on the number of runs needed for the convergence of the empirical average to the expected reward uniformly for a class of policies, in terms of the V-C or pseudo dimension of the policy class. We then propose a framework to obtain an-optimal policy from simulation. We provide sample complexity of such an approach.

Symbolic and Quantitative Approaches to Reasoning with Uncertainty, 2017

Possibilistic Markov Decision Processes offer a compact and tractable way to represent and solve problems of sequential decision under qualitative uncertainty. Even though appealing for its ability to handle qualitative problems, this model suffers from the drowning effect that is inherent to possibilistic decision theory. The present paper proposes to escape the drowning effect by extending to stationary possibilistic MDPs the lexicographic preference relations defined in [6] for non-sequential decision problems and provides a value iteration algorithm to compute policies that are optimal for these new criteria.

Annals of Mathematics and Artificial Intelligence, 2010

An optimal probabilistic-planning algorithm solves a problem, usually modeled by a Markov decision process, by finding an optimal policy. In this paper, we study the k best policies problem. The problem is to find the k best policies of a discrete Markov decision process. The k best policies, k > 1, cannot be found directly using dynamic programming. Naïvely, finding the k-th best policy can be Turing reduced to the optimal planning problem, but the number of problems queried in the naïve algorithm is exponential in k. We show empirically that solving k best policies problem by using this reduction requires unreasonable amounts of time even when k = 3. We then provide two new algorithms. The first is a complete algorithm, based on our theoretical contribution that the k-th best policy differs from the i-th policy, for some i < k, on exactly one state. The second is an approximate algorithm that skips many less useful policies. We show that both algorithms have good scalability. We also show that the approximate algorithms runs much faster and finds interesting, high-quality policies.

IEEE Transactions on Automatic Control, 2005

We propose a novel algorithm called Evolutionary Policy Iteration (EPI) for solving in nite horizon discounted reward Markov Decision Process (MDP) problems. EPI inherits the spirit of the well-known PI algorithm but eliminates the need to maximize over the entire action space in the policy improvement step, so it should be most e ective for problems with very large action spaces. EPI iteratively generates a \population" or a set of policies such that the performance of the \elite policy" for a population is monotonically improved with respect to a de ned tness function. EPI converges with probability one to a population whose elite policy is an optimal policy for a given MDP. EPI is naturally parallelizable and along this discussion, a distributed variant of PI is also studied.

This letter investigates the structure of the optimal policy for a class of Markov decision processes (MDPs), having convex and piecewise linear cost function. The optimal policy is proved to have a piecewise linear structure that alternates flat and constant-slope pieces, resembling a staircase with tilted rises and as many steps (thresholds) as the breakpoints of the cost function. This particular structure makes it possible to express the policy in a very compact manner, particularly suitable to be stored in low-end devices. More importantly, the thresholdbased form of the optimal policy can be exploited to reduce the computational complexity of the iterative dynamic programming (DP) algorithm used to solve the problem. These results apply to a rather wide set of optimization problems, typically involving the management of a resource buffer such as the energy stored in a battery, or the packets queued in a wireless node.

Siam Journal on Control and Optimization, 1993

This work is a survey of the average cost control problem for discrete-time Markov processes. The authors have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades. The exposition ranges from finite to Borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies. The authors have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography. The authors have also identified several important questions that are still open to investigation. 282 discrete-time controlled markov processes 2.1. The model. A discrete-time, stationary controlled Markov process (CMP), or Markov decision process, is a stochastic dynamical system specified by the five-tuple S, A, U, P, c , where (a) S is a Borel space, called the state space, the elements of which are called states;

2017 Brazilian Conference on Intelligent Systems (BRACIS)