Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Solution Methods
Related Work
Analysis
Synthetic MDPS: Best and Worst Cases
Discussion
References
All Topics
Computer Science
Machine Learning

Stochastic dynamic programming with factored representations

Profile image of Craig BoutilierCraig Boutilier

2000, Artificial Intelligence

Abstract

Markov decision processes (MDPs) have proven to be popular models for decision-theoretic planning, but standard dynamic programming algorithms for solving MDPs rely on explicit, state-based specifications and computations. To alleviate the combinatorial problems associated with such methods, we propose new representational and computational techniques for MDPs that exploit certain types of problem structure. We use dynamic Bayesian networks (with decision trees representing the local families of conditional probability distributions) to represent stochastic actions in an MDP, together with a decision-tree representation of rewards. Based on this representation, we develop versions of standard dynamic programming algorithms that directly manipulate decision-tree representations of policies and value functions. This generally obviates the need for state-by-state computation, aggregating states at the leaves of these trees and requiring computations only for each aggregate state. The key to these algorithms is a decision-theoretic generalization of classic regression analysis, in which we determine the features relevant to predicting expected value. We demonstrate the method empirically on several planning problems, showing significant savings for certain types of problems. We also identify certain classes of problems for which this technique fails to perform well and suggest extensions and related ideas that may prove useful for such problems. We also briefly describe an approximation scheme based on this approach.

Related papers

Probabilistic planning via linear value-approximation of first-order MDPs
Craig Boutilier

2005

Abstract We describe a probabilistic planning approach that translates a PPDDL planning problem description to a first-order MDP (FOMDP) and uses approximate solution techniques for FOMDPs to derive a value function and corresponding policy. Our FOMDP solution techniques represent the value function linearly wrt a set of first-order basis functions and compute suitable weights using lifted, first-order extensions of approximate linear programming (FOALP) and approximate policy iteration (FOAPI) for MDPs.

View PDFchevron_right
Symbolic Dynamic Programming for Discrete and Continuous State MDPs
Leliane Nunes de Barros

2011

Many real-world decision-theoretic planning problems can be naturally modeled with discrete and continuous state Markov decision processes (DC-MDPs). While previous work has addressed automated decision-theoretic planning for DC-MDPs, optimal solutions have only been defined so far for limited settings, e.g., DC-MDPs having hyper-rectangular piecewise linear value functions. In this work, we extend symbolic dynamic programming (SDP) techniques to provide optimal solutions for a vastly expanded class of DC-MDPs. To address the inherent combinatorial aspects of SDP, we introduce the XADD-a continuous variable extension of the algebraic decision diagram (ADD)-that maintains compact representations of the exact value function. Empirically, we demonstrate an implementation of SDP with XADDs on various DC-MDPs, showing the first optimal automated solutions to DC-MDPs with linear and nonlinear piecewise partitioned value functions and showing the advantages of constraint-based pruning for XADDs.

View PDFchevron_right
Approximating value trees in structured dynamic programming
Craig Boutilier

MACHINE LEARNING-INTERNATIONAL …, 1996

We propose and examine a method of approximate dynamic programming for Markov decision processes based on structured problem representations. We assume an MDP is represented using a dynamic Bayesian network, and construct value functions using decision trees as our function representation. The size of the representation is kept within acceptable limits by pruning these value trees so that leaves represent possible ranges of values, thus approximating the value functions produced during optimization. We propose a method for detecting convergence,prove errors bounds on the resulting approximately optimal value functions and policies, and describe some preliminary experimental results.

View PDFchevron_right
Decision-Theoretic Planning with non-Markovian Rewards
Charles Gretton, J. Slaney

Journal of Artificial Intelligence Research, 2006

A decision process in which rewards depend on history rather than merely on the current state is called a decision process with non-Markovian rewards (NMRDP). In decisiontheoretic planning, where many desirable behaviours are more naturally expressed as properties of execution sequences rather than as properties of states, NMRDPs form a more natural model than the commonly adopted fully Markovian decision process (MDP) model. While the more tractable solution methods developed for MDPs do not directly apply in the presence of non-Markovian rewards, a number of solution methods for NMRDPs have been proposed in the literature. These all exploit a compact specification of the non-Markovian reward function in temporal logic, to automatically translate the NMRDP into an equivalent MDP which is solved using efficient MDP solution methods. This paper presents nmrdpp(Non-Markovian Reward Decision Process Planner), a software platform for the development and experimentation of methods for decision-theoretic planning with non-Markovian rewards. The current version of nmrdpp implements, under a single interface, a family of methods based on existing as well as new approaches which we describe in detail. These include dynamic programming, heuristic search, and structured methods. Using nmrdpp, we compare the methods and identify certain problem features that affect their performance. nmrdpp's treatment of non-Markovian rewards is inspired by the treatment of domain-specific search control knowledge in the TLPlan planner, which it incorporates as a special case. In the First International Probabilistic Planning Competition, nmrdpp was able to compete and perform well in both the domain-independent and hand-coded tracks, using search control knowledge in the latter.

View PDFchevron_right
NMRDPP: A System for Decision-Theoretic Planning with Non-Markovian Rewards
Charles Gretton

This paper examines a number of solution methods for decision processes with non-Markovian rewards (NMRDPs). They all exploit a temporal logic specification of the reward function to automatically translate the NMRDP into an equivalent Markov decision process (MDP) amenable to well-known MDP solution methods. They differ however in the representation of the target MDP and the class of MDP solution methods to which they are suited. As a result, they adopt different temporal logics and different translations. Unfortunately, no implementation of these methods nor experimental let alone comparative results have ever been reported. This paper is the first step towards filling this gap. We describe an integrated system for solving NMRDPs which implements these methods and several variants under a common interface; we use it to compare the various approaches and identify certain problem features favouring one over the other.

View PDFchevron_right
Simulation-based Algorithms for Markov Decision Processes
Michael Fu

Communications and Control Engineering, 2007

View PDFchevron_right
Risk-Sensitive Planning with Probabilistic Decision Graphs
Reid Simmons

Principles of Knowledge Representation and Reasoning, 1994

Probabilistic AI planning methods that minimize expected execution cost have a neutral attitude towards risk. We demonstrate how one can transform planning problems for risk-sensitive agents into equivalent ones for risk-neutral agents provided that exponential utility functions are used. The transformed planning problems can then be solved with these existing AI planning methods. To demonstrate our ideas, we use a probabilistic planning framework probabilistic decision graphs" that can easily be mapped into Markov decision problems. It allows one to describe probabilistic e ects of actions, actions with di erent costs resource consumption, and goal states with di erent rewards. We show the use of probabilistic decision graphs for nding optimal plans for risk-sensitive agents in a stochastic blocksworld domain.

View PDFchevron_right
Exploiting structure in policy construction
Craig Boutilier

International Joint Conference on …, 1995

Markov decision processes (MDPs) have recently been applied to the problem of modeling decision-theoretic planning. While such traditional methods for solving MDPs are often practical for small states spaces, their effectiveness for large AI planning problems is questionable. We present an algorithm, called structured policy iteration (SPI), that constructs optimal policies without explicit enumeration of the state space. The algorithm retains the fundamental computational steps of the commonly used modified policy iteration algorithm, but exploits the variable and propositional independencies in reflected in a temporal Bayesian network representation of MDPs. The principles behind SPI can be applied to any structured representation of stochastic actions, and the algorithm itself can be used in conjunction with recent approximation methods.

View PDFchevron_right
NMRDPP: Decision-Theoretic Planning with Control Knowledge
Charles Gretton

We discuss NMRDPP, a system for solving decision processes with non-Markovian reward. More specifically, target decision processes exhibit Markovian dynamics and rewarding behaviours are modelled as state trajectories specified in a linear temporal logic. In addition to implementing structured, tabular and online MDP solution algorithms, NMRDPP can exploit domain specific control knowledge. State trajectories which violate the users knowledge/intuition regarding useful dynamics can be pruned from consideration by the MDP solution algorithm. Thus, in addition to facilitating concise specification of complex reward structures, NMRDPP can be used to greatly speed up policy computation for propositional MDPs. To our knowledge, NMRDPP is the only implementation of solution algorithms designed to solve decision processes with non-Markovian rewards.

View PDFchevron_right
Planning With Agents: An Efficient Approach Using Hierarchical Dynamic Decision Networks
John Rose

To be useful in solving real world problems, agents need to be able to act in environments in which it may not be possible to be completely aware of the current state and where actions do not always work as planned. Additional complexity is added to the problem when one considers groups of agents working together. By casting the agent planning problem as a partially observable Markov decision problem (POMDP), optimal policies can be generated for partially observable and stochastic environments. Exact solutions, however, are notoriously difficult to find for problems of a realistic nature. We introduce a hierarchical decision network-based planning algorithm that can generate high quality plans during execution while demonstrating significant time savings. We also discuss how this approach is particularly applicable to planning in a multiagent environment as compared to other POMDP-based planning algorithms. We present experimental results comparing our algorithm with results obtain...

View PDFchevron_right

References (84)

  1. References [1] Christopher G. Atkeson, Andrew W. Moore, and Stefan Schaal. Locally weighted learning for control. Artificial Intelligence Review, 11:75-113, 1997.
  2. R. Iris Bahar, Erica A. Frohm, Charles M. Gaona, Gary D. Hachtel, Enrico Macii, Abelardo Pardo, and Fabio Somenzi. Algebraic decision diagrams and their applications. In International Conference on Computer-Aided Design, pages 188-191. IEEE, 1993.
  3. Richard E. Bellman. Dynamic Programming. Princeton University Press, Princeton, 1957.
  4. D. P. Bertsekas and D. A. Castanon. Adaptive aggregation for infinite horizon dynamic programming. IEEE Transactions on Automatic Control, 34:589-598, 1989.
  5. Dimitri P. Bertsekas. Dynamic Programming: Deterministic and Stochastic Models. Prentice-Hall, Englewood Cliffs, 1987.
  6. Dimitri P. Bertsekas and John. N. Tsitsiklis. Neuro-dynamic Programming. Athena, Belmont, MA, 1996.
  7. Marko Bohanic and Ivan Bratko. Trading accuracy for simplicity in decision trees. Machine Learning, 15:223- 250, 1994.
  8. Craig Boutilier. Correlated action effects in decision theoretic regression. In Proceedings of the Thirteenth Con- ference on Uncertainty in Artificial Intelligence, pages 30-37, Providence, RI, 1997.
  9. Craig Boutilier, Ronen I. Brafman, and Christopher Geib. Prioritized goal decomposition of Markov decision processes: Toward a synthesis of classical and decision theoretic planning. In Proceedings of the Fifteenth Inter- national Joint Conference on Artificial Intelligence, pages 1156-1162, Nagoya, 1997.
  10. Craig Boutilier, Ronen I. Brafman, and Christopher Geib. Structured reachability analysis for Markov decision processes. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, pages 24-32, Madison, WI, 1998.
  11. Craig Boutilier, Thomas Dean, and Steve Hanks. Decision theoretic planning: Structural assumptions and com- putational leverage. Journal of Artificial Intelligence Research, 11:1-94, 1999.
  12. Craig Boutilier and Richard Dearden. Using abstractions for decision-theoretic planning with time constraints. In Proceedings of the Twelfth National Conference on Artificial Intelligence, pages 1016-1022, Seattle, 1994.
  13. Craig Boutilier and Richard Dearden. Approximating value trees in structured dynamic programming. In Pro- ceedings of the Thirteenth International Conference on Machine Learning, pages 54-62, Bari, Italy, 1996.
  14. Craig Boutilier, Nir Friedman, Moisés Goldszmidt, and Daphne Koller. Context-specific independence in Bayesian networks. In Proceedings of the Twelfth Conference on Uncertainty in Artificial Intelligence, pages 115-123, Portland, OR, 1996.
  15. Craig Boutilier and Moisés Goldszmidt. The frame problem and Bayesian network action representations. In Proceedingsof the Eleventh Biennial Canadian Conference on Artificial Intelligence, pages 69-83, Toronto, 1996.
  16. Craig Boutilier and David Poole. Computing optimal policies for partially observable decision processes using compact representations. In Proceedings of the Thirteenth National Conference on Artificial Intelligence, pages 1168-1175, Portland, OR, 1996.
  17. Craig Boutilier and Martin L. Puterman. Process-oriented planning and average-reward optimality. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pages 1096-1103, Montreal, 1995.
  18. Justin A. Boyan and Andrew W. Moore. Generalization in reinforcement learning: Safely approximating the value function. In G. Tesauro, D. S. Touretzky, and T. K. Leen, editors, Advances in Neural Information Processing Systems 7. MIT Press, Cambridge, 1995.
  19. Randal E. Bryant. Graph-based algorithms for boolean function manipulation. IEEE Transactions on Computers, C-35(8):677-691, 1986.
  20. J. R. Burch, E. M. Clarke, K. L. McMillan, D. L. Dill, and L. J. Hwang. Symbolic model checking: 10 20 states and beyond. In Conference on Logic in Computer Science, pages 428-439, 1990.
  21. Anthony R. Cassandra, Leslie Pack Kaelbling, and Michael L. Littman. Acting optimally in partially observable stochastic domains. In Proceedings of the Twelfth National Conference on Artificial Intelligence, pages 1023- 1028, Seattle, 1994.
  22. David Chapman. Planning for conjunctive goals. Artificial Intelligence, 32(3):333-377, 1987.
  23. David Chapman and Leslie Pack Kaelbling. Input generalization in delayed reinforcement learning: An algo- rithm and performance comparisons. In Proceedings of the Twelfth International Joint Conference on Artificial Intelligence, pages 726-731, Sydney, 1991.
  24. E. M. Clarke, E. A. Emerson, and A.P. Sistla. Automatic verification of finite state concurrent systems using temporal logic specifications: A practical approach. In Symposium on Principles of Programming Languages, pages 117-126. ACM, 1983.
  25. Adnan Darwiche and Moisés Goldszmidt. Action networks: A framework for reasoning about actions and change under uncertainty. In Proceedings of the Tenth Conference on Uncertainty in Artificial Intelligence, pages 136- 144, Seattle, 1994.
  26. Thomas Dean and Robert Givan. Model minimization in Markov decision processes. In Proceedings of the Four- teenth National Conference on Artificial Intelligence, pages 106-111, Providence, 1997.
  27. Thomas Dean, Robert Givan, and Sonia Leach. Model reduction techniques for computing approximately optimal solutions for Markov decision processes. In Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence, pages 124-131, Providence, RI, 1997.
  28. Thomas Dean, Leslie Pack Kaelbling, Jak Kirman, and Ann Nicholson. Planning with deadlines in stochastic do- mains. In Proceedingsof the Eleventh National Conferenceon Artificial Intelligence, pages 574-579, Washington, D.C., 1993.
  29. Thomas Dean and Keiji Kanazawa. A model for reasoning about persistence and causation. Computational In- telligence, 5(3):142-150, 1989.
  30. Richard Dearden and Craig Boutilier. Abstraction and approximate decision theoretic planning. Artificial Intelli- gence, 89:219-283, 1997.
  31. Thomas G. Dietterich. The MAXQ method for hierarchical reinforcement learning. In Proceedingsof the Fifteenth International Conference on Machine Learning, pages 118-126, Madison, WI, 1998.
  32. Thomas G. Dietterich and Nicholas S. Flann. Explanation-based learning and reinforcement learning: A unified approach. In Proceedings of the Twelfth International Conference on Machine Learning, pages 176-184, Lake Tahoe, 1995.
  33. Thomas G. Dietterich and Nicholas S. Flann. Explanation-based learning and reinforcement learning: A unified view. Machine Learning, 28(2):169-210, 1997.
  34. Edsger W. Dijkstra. Guarded commands, nondeterminacy and formal derivation of programs. Communications of the ACM, 18(8):453-457, 1975.
  35. Jerome A. Feldman and Robert F. Sproull. Decision theory and artificial intelligence II: The hungry monkey. Cognitive Science, 1:158-192, 1977.
  36. Richard E. Fikes and Nils J. Nilsson. STRIPS: A new approach to the application of theorem proving to problem solving. Artificial Intelligence, 2:189-208, 1971.
  37. Zoltán Gábor, Zsolt Kalmár, and Csaba Szepesvári. Multi-criteria reinforcement learning. In Proceedings of the Fifteenth International Conference on Machine Learning, pages 197-205, Madison, WI, 1998.
  38. Dan Geiger and David Heckerman. Advances in probabilistic reasoning. In Proceedingsof the Seventh Conference on Uncertainty in Artificial Intelligence, pages 118-126, Los Angeles, 1991.
  39. Robert Givan and Thomas Dean. Model minimization, regression, and propositional STRIPS planning. In Pro- ceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pages 1163-1168, Nagoya, Japan, 1997.
  40. Steve Hanks and Drew V. McDermott. Modeling a dynamic and uncertain world i: Symbolic and probabilistic reasoning about change. Artificial Intelligence, 1994.
  41. Steven John Hanks. Projecting Plans for Uncertain Worlds. PhD thesis, Yale University, 1990.
  42. J. Hartmanis and R. E. Stearns. Algebraic Structure Theory of Sequential Machines. Prentice-Hall, Englewood Cliffs, 1966.
  43. Jesse Hoey, Robert St-Aubin, Alan Hu, and Craig Boutilier. SPUDD: Stochastic planning using decision diagrams. In Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence, pages 279-288, Stockholm, 1999.
  44. Ronald A. Howard. Dynamic Programming and Markov Processes. MIT Press, Cambridge, 1960.
  45. Ronald A. Howard and James E. Matheson, editors. Readings on the Principles and Applications of Decision Analysis. Strategic Decision Group, Menlo Park, CA, 1984.
  46. L. Hyafil and R. L. Rivest. Constructing optimal binary decision trees is NP-complete. Information Processing Letters, 5:15-17, 1976.
  47. Leslie Pack Kaelbling. Hierarchical reinforcement learning: Preliminary results. In Proceedings of the Tenth International Conference on Machine Learning, pages 167-173, Amherst, MA, 1993.
  48. Leslie Pack Kaelbling, Michael L. Littman, and Andrew W. Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4:237-285, 1996.
  49. R. L. Keeney and H. Raiffa. Decisions with Multiple Objectives: Preferences and Value Trade-offs. Wiley, New York, 1976.
  50. Nicholas Kushmerick, Steve Hanks, and Daniel Weld. An algorithm for probabilistic planning. Artificial Intelli- gence, 76:239-286, 1995.
  51. D. Lee and M. Yannakakis. Online miminization of transition systems. In Proceedings of the 24th Annual ACM Symposium on the Theory of Computing (STOC-92), pages 264-274, Victoria, BC, 1992.
  52. Michael Lederman Littman. Algorithms for sequential decision making. Ph.D. Thesis CS-96-09, Brown Uni- versity, Department of Computer Science, March 1996.
  53. William S. Lovejoy. A survey of algorithmic methods for partially observed Markov decision processes. Annals of Operations Research, 28:47-66, 1991.
  54. David McAllester and David Rosenblitt. Systematic nonlinear planning. In Proceedings of the Ninth National Conference on Artificial Intelligence, pages 634-639, Anaheim, 1991.
  55. John McCarthy and P.J. Hayes. Some philosophical problems from the standpoint of artificial intelligence. Ma- chine Intelligence, 4:463-502, 1969.
  56. Nicolas Meuleau, Milos Hauskrecht, Kee-Eung Kim, Leonid Peshkin, Leslie Pack Kaelbling, Thomas Dean, and Craig Boutilier. Solving very large weakly coupled Markov decision processes. In Proceedings of the Fifteenth National Conference on Artificial Intelligence, pages 165-172, Madison, WI, 1998.
  57. Judea Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Mateo, 1988.
  58. J. Scott Penberthy and Daniel S. Weld. UCPOP: A sound, complete, partial order planner for adl. In Proceedings of the Third International Conference on Principles of Knowledge Representation and Reasoning, pages 103-114, Cambridge, MA, 1992.
  59. David Poole. Probabilistic Horn abduction and Bayesian networks. Artificial Intelligence, 64(1):81-129, 1993.
  60. David Poole. The independentchoice logic for modelling multiple agents under uncertainty. Artificial Intelligence, 94(1-2):7-56, 1997.
  61. Doina Precup, Richard S. Sutton, and Satinder Singh. Theoretical results on reinforcement learning with tempo- rally abstract behaviors. In Proceedings of the Tenth European Conference on Machine Learning, pages 382-393, Chemnitz, Germany, 1998.
  62. Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York, 1994.
  63. Martin L. Puterman and M.C. Shin. Modified policy iteration algorithms for discounted Markov decision prob- lems. Management Science, 24:1127-1137, 1978.
  64. J. Ross Quinlan. C45: Programs for Machince Learning. Morgan Kaufmann, San Mateo, 1993.
  65. Ronald L. Rivest. Learning decision lists. Machine Learning, 2:229-246, 1987.
  66. Earl D. Sacerdoti. The nonlinear nature of plans. In Proceedings of the Fourth International Joint Conference on Artificial Intelligence, pages 206-214, 1975.
  67. M. J. Schoppers. Universal plans for reactive robots in unpredictable environments. In Proceedings of the Tenth International Joint Conference on Artificial Intelligence, pages 1039-1046, Milan, 1987.
  68. Paul L. Schweitzer, Martin L. Puterman, and Kyle W. Kindle. Iterative aggregation-disaggregation procedures for discounted semi-Markov reward processes. Operations Research, 33:589-605, 1985.
  69. Ross D. Shachter. Evaluating influence diagrams. Operations Research, 33(6):871-882, 1986.
  70. Solomon E. Shimony. The role of relevance in explanation I: Irrelevance as statistical independence. International Journal of Approximate Reasoning, 8(4):281-324, 1993.
  71. Satinder P. Singh and David Cohn. How to dynamically merge Markov decision processes. In Advances in Neural Information Processing Systems 10, pages 1057-1063. MIT Press, Cambridge, 1998.
  72. Satinder Pal Singh. Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning, 8:323-339, 1992.
  73. Richard D. Smallwood and Edward J. Sondik. The optimal control of partially observable Markov processes over a finite horizon. Operations Research, 21:1071-1088, 1973.
  74. James E. Smith, Samuel Holtzman, and James E. Matheson. Structuring conditional relationships in influence diagrams. Operations Research, 41(2):280-297, 1993.
  75. Edward J. Sondik. The optimal control of partially observable Markov processes over the infinite horizon: Dis- counted costs. Operations Research, 26:282-304, 1978.
  76. Richard S. Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proceedings of the Seventh International Conference on Machine Learning, pages 216-224, Austin, 1990.
  77. Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 1998.
  78. Jonathan Tash and Stuart Russell. Control strategies for a stochastic planner. In Proceedings of the Twelfth Na- tional Conference on Artificial Intelligence, pages 1079-1085, Seattle, 1994.
  79. Joseph A. Tatman and Ross D. Shachter. Dynamic programming and influence diagrams. IEEE Transactions on Systems, Man and Cybernetics, 20(2):365-379, 1990.
  80. Gerald J. Tesauro. TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural Com- putation, 6:215-219, 1994.
  81. John H. Tsitsiklis and Benjamin Van Roy. Feature-based methods for large scale dynamic programming. Machine Learning, 22:59-94, 1996.
  82. Paul E. Utgoff. Decision tree induction based on efficient tree restructuring. Technical Report 95-18, University of Massachusetts, March 1995.
  83. Richard Waldinger. Achieving several goals simultaneously. In E. Elcock and D. Mitchie, editors, Machine In- telligence 8: Machine Representations of Knowledge, pages 94-136. Ellis Horwood, Chichester, England, 1977.
  84. Christopher J. C. H. Watkins and Peter Dayan. Q-learning. Machine Learning, 8:279-292, 1992.

Related papers

Solving Hybrid Markov Decision Processes
Eduardo Morales

Lecture Notes in Computer Science, 2006

Markov decision processes (MDPs) have developed as a standard for representing uncertainty in decision-theoretic planning. However, MDPs require an explicit representation of the state space and the probabilistic transition model which, in continuous or hybrid continuousdiscrete domains, are not always easy to define. Even when this representation is available, the size of the state space and the number of state variables to consider in the transition function may be such that the resulting MDP cannot be solved using traditional techniques. In this paper a reward-based abstraction for solving hybrid MDPs is presented. In the proposed method, we gather information about the rewards and the dynamics of the system by exploring the environment. This information is used to build a decision tree (C4.5) representing a small set of abstract states with equivalent rewards, and then is used to learn a probabilistic transition function using a Bayesian networks learning algorithm (K2). The system output is a problem specification ready for its solution with traditional dynamic programming algorithms. We have tested our abstract MDP model approximation in real-world problem domains. We present the results in terms of the models learned and their solutions for different configurations showing that our approach produces fast solutions with satisfying policies.

View PDFchevron_right
SPUDD: Stochastic planning using decision diagrams
Craig Boutilier

Proceedings of the Fifteenth …, 1999

Recently, structured methods for solving factored Markov decisions processes (MDPs) with large state spaces have been proposed recently to allow dynamic programming to be applied without the need for complete state enumeration. We propose and examine a new value iteration algorithm for MDPs that uses algebraic decision diagrams (ADDs) to represent value functions and policies, assuming an ADD input representation of the MDP. Dynamic programming is implemented via ADD manipulation. We demonstrate our method on a class of large MDPs (up to 63 million states) and show that significant gains can be had when compared to tree-structured representations (with up to a thirty-fold reduction in the number of nodes required to represent optimal value functions).

View PDFchevron_right
FASTER DYNAMIC PROGRAMMING FOR MARKOV DECISION PROCESSES
Peng Dai

2007

Markov decision processes (MDPs) are a general framework used in artificial intelligence (AI) to model decision theoretic planning problems. Solving real world MDPs has been a major and challenging research topic in the AI literature, since classical dynamic programming algorithms converge slowly. We discuss two approaches in expediting dynamic programming. The first approach combines heuristic search strategies with dynamic programming to expedite the convergence process. The second makes use of graphical structures in MDPs to perform dynamic programming in a better order.

View PDFchevron_right
Planning and programming with first-order markov decision processes: insights and challenges
Craig Boutilier

2001

Abstract Markov decision processes (MDPs) have become the de facto standard model for decision-theoretic planning problems. However, classic dynamic programming algorithms for MDPs [22] require explicit state and action enumeration. For example, the classical representation of a value function is a table or vector associating a value with each system state; such value functions are produced by iterating over the state space.

View PDFchevron_right
Decision-theoretic planning: Structural assumptions and computational leverage
Craig Boutilier, Cs Chair Tom Dean

Journal of Artificial Intelligence Research, 1999

Planning under uncertainty is a central problem in the study of automated sequential decision making, and has been addressed by researchers in many different fields, including AI planning, decision analysis, operations research, control theory and economics. While the assumptions and perspectives adopted in these fields often differ in substantial ways, many planning problems of interest to researchers in these fields can be modeled as Markov decision processes (MDPs) and analyzed using the techniques of decision theory. This paper presents an overview and synthesis of MDP-related methods showing how they provide a unifying framework for modeling many classes of planning problems studied in AI. It also describes structural properties of MDPs that, when exhibited by particular classes of problems, can be exploited in the construction of optimal or approximately optimal policies or plans. Planning problems commonly possess structure in the reward and value functions used to describe performance criteria, in the functions used to describe state transitions and observations, and in the relationships among features used to describe states, actions, rewards, and observations. Specialized representations, and algorithms employing these representations, can achieve computational leverage by exploiting these various forms of structure. Certain AI techniques---in particular those based on the use of structured, intensional representations---can be viewed in this way. This paper surveys several types of representations for both classical and decision theoretic planning problems, and planning algorithms that exploit these representations in a number of different ways to ease the computational burden of constructing policies or plans. It focuses primarily on abstraction, aggregation and decomposition techniques based on AI-style representations.

View PDFchevron_right

Related topics

  • Cognitive Science
  • Artificial Intelligence
  • Markov Decision Process
  • Probability Distribution & Appli...
  • Bayesian Network
  • dynamic Bayesian network
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved