Markov Decision Processes

This paper proposes a novel and practical model-based learning approach with iterative refinement for solving continuous (and hybrid) Markov decision processes. Initially, an approximate model is learned using conventional sampling methods and solved to obtain a policy. Iteratively, the approximate model is refined using variance in the utility values as partition criterion. In the learning phase, initial reward and transition functions are obtained by sampling the state-action space. The samples are used to induce a decision tree predicting reward values from which an initial partition of the state space is built. The samples are also used to induce a factored MDP. The state abstraction is then refined by splitting states only where the split is locally important. The main contributions of this paper are the use of sampling to construct an abstraction, and a local refinement process of the state abstraction based on utility variance. The proposed technique was tested in AsistO, an ...

We study the problem of long-run average cost control of Markov chains conditioned on a rare event. In a related recent work, a simulation based algorithm for estimating performance measures associated with a Markov chain conditioned on a rare event has been developed. We extend ideas from this work and develop an adaptive algorithm for obtaining, online, optimal control policies conditioned on a rare event. Our algorithm uses three timescales or step-size schedules. On the slowest timescale, a gradient search algorithm for policy updates that is based on one-simulation simultaneous perturbation stochastic approximation (SPSA) type estimates is used. Deterministic perturbation sequences obtained from appropriate normalized Hadamard matrices are used here. The fast timescale recursions compute the conditional transition probabilities of an associated chain by obtaining solutions to the multiplicative Poisson equation (for a given policy estimate). Further, the risk parameter associated with the value function for a given policy estimate is updated on a timescale that lies in between the two scales above. We briefly sketch the convergence analysis of our algorithm and present a numerical application in the setting of routing multiple flows in communication networks.

This is a study of simple random walks, birth and death processes, and M/M/s queues that have transition probabilities and rates that are sequentially controlled at jump times of the processes. Each control action yields a one-step reward depending on the chosen probabilities or transition rates and the state of the process. The aim is to find control policies that maximize the total discounted or average reward. Conditions are given for these processes to have certain natural monotone optimal policies. Under such a policy for the M/M/s queue, for example, the service and arrival rates are non-decreasing and non-increasing functions, respectively, of the queue length. Properties of these policies and a linear program for computing them are also discussed.

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

The paper studies the problem of allocating bandwidth resources of a Service Overlay Network, to optimize revenue. Clients bid for network capacity in periodically held auctions, under the condition that resources allocated in an auction are reserved for the entire duration of the connection, not subject to future contention. This makes the optimal allocation coupled over time, which we formulate as a Markov Decision Process (MDP). Studying first the single resource case, we develop a receding horizon approximation to the optimal MDP policy, using current revenue and the expected revenue in the next step to make bandwidth assignments. A second approximation is then found, suitable for generalization to the network case, where bids for different routes compete for shared resources. In that case we develop a distributed implementation of the auction, and demonstrate its performance through simulations.

From the reviews:" The authors of this book, part of the Medical Radiology-Diagnostic Imaging and radiation Oncology series, present a sound approach to diagnosis of lung disease based on the CT appearance of the pulmonary parenchyma.... The book is appropriate for the practicing radiologist who wishes to embark on evaluation of the lungs and their diseases according to anatomic parameters, especially HRCT.... Clinical physicians with a particular interest in pulmonary disease will find the book useful as well." ...

The problem of solving large Markov decision processes accurately and quickly is challenging. Since the computational effort incurred is considerable, current research focuses on finding superior acceleration techniques. For instance, the convergence properties of current solution methods depend, to a great extent, on the order of backup operations. On one hand, algorithms such as topological sorting are able to find good orderings but their overhead is usually high. On the other hand, shortest path methods, such as Dijkstra's algorithm which is based on priority queues, have been applied successfully to the solution of deterministic shortest-path Markov decision processes. Here, we propose an improved value iteration algorithm based on Dijkstra's algorithm for solving shortest path Markov decision processes. The experimental results on a stochastic shortest-path problem show the feasibility of our approach.

In reinforcement learning an agent uses online feedback from the environment and prior knowledge in order to adaptively select an effective policy. Model free approaches address this task by directly mapping external and internal states to actions, while model based methods attempt to construct a model of the environment, followed by a selection of optimal actions based on that model. Given the complementary advantages of both approaches, we suggest a novel algorithm which combines them into a single algorithm, which switches between a model based and a model free mode, depending on the current environmental state and on the status of the agent's knowledge. We prove that such an approach leads to improved performance whenever environmental knowledge is available, without compromising performance when such knowledge is absent. Numerical simulations demonstrate the effectiveness of the approach and suggest its efficacy in boosting policy gradient learning.

Possibilistic de cision theory has be e n propose d twe nty years ago and has had se ve ral extensions since the n. Even though ap pe aling for its ability to handle qualitative decision proble ms, possibilistic decision the ory suffe rs from an important drawback. Qualitative possibilistic utility criteria compare acts through min and max operators, which Jeads to a drowning effect. To over come this Jack of decision power of the theory, several refinements have been proposed. Lexicographie refinements are particularly appealing since they allow to benefit from the Expected Utility background, while remaining qualitative. This article aims at extend ing lexicographie refinements to sequential decision problems i.e., to possibilistic decision trees and possibilistic Markov decision proce sse s, whe n the horizon is finite . We present two crite ria that re fine qualitative possibilistic utilitie s and provide dynarnic prograrnming algorithrns for calculating Jexicographically optimal policies.

In this paper we will focus on spatialized decision problems which we propose to model in the framework of (highly) multidimensional Markov Decision Processes (MDPs) which exhibit only local dependencies between variables. We propose to approximate a Markov chain on a multidimensional random variable by a Markov chain on a set of weakly dependent random variables. This allows to (approximately) solve multidimensional MDPs with hundreds of variables, to the price of a loss of exactness of the process model. The method is mostly empirical yet, however it allows to deal with decision problems far larger than the one usually dealt with in the MDP framework.

In this letter, it is shown that the randomized shortest-path framework (RSP, [15]) provides a theoretical interpretation of a class of ant colony optimization (ACO) algorithms, enjoying some nice properties. According to RSP, ants are sent from some initial node until they either eventually reach the goal node, or abandon and come back unsuccessfully along the same path. During their return travel (backward pass), each node on the trajectory is rewarded if the goal was reached-successful walk. The policy, which takes the form of the probabilities of following arc k → k in each node k, is updated periodically at each epoch t, and is set to the previous policy times (1) the proportion of successful walks starting from node k (probability of success, the pheromone), and (2) exp[−θc kk ] (the heuristic function), where c kk is the cost associated to arc k → k. The RSP framework shows that (i) this policy is optimal at any epoch t in that it minimizes the expected cost for reaching the goal node (exploitation) while maintaining a constant relative entropy spread in the graph (exploration), and (ii) the procedure converges to the minimal cost policy when t → ∞, provided the probability of success is well-estimated, that is, enough ants are sent at each epoch (asymptotic convergence). In other words, it provides an optimal trade-off between exploration and exploitation. We therefore decided to bring the RSP framework to the attention of the evolutionary computation community, hoping that it will stimulate the design as well as the empirical evaluation of new ACO algorithms having interesting theoretical properties.

Ye showed recently that the simplex method with Dantzig pivoting rule, as well as Howard's policy iteration algorithm, solve discounted Markov decision processes (MDPs), with a constant discount factor, in strongly polynomial time. More precisely, Ye showed that both algorithms terminate after at most O mn 1−γ log n 1−γ

We study discrete-time discounted constrained Markov decision processes (CMDPs) with Borel state and action spaces. These CMDPs satisfy either weak (W) continuity conditions, that is, the transition probability is weakly continuous and the reward function is upper semicontinuous in state-action pairs, or setwise (S) continuity conditions, that is, the transition probability is setwise continuous and the reward function is upper semicontinuous in actions. Our main goal is to study models with unbounded reward functions, which are often encountered in applications, e.g., in consumption/investment problems. We provide some general assumptions under which the optimization problems in CMDPs are solvable in the class of randomized stationary policies and in the class of chattering policies introduced in this paper. If the initial distribution and transition probabilities are atomless, then using a general "purification result" of Feinberg and Piunovskiy we show the existence of a deterministic (stationary) optimal policy. Our main results are illustrated by examples.

In this paper, we study algorithms for special cases of energy games, a class of turn-based games on graphs that show up in the quantitative analysis of reactive systems. In an energy game, the vertices of a weighted directed graph belong either to Alice or to Bob. A token is moved to a next vertex by the player controlling its current location, and its energy is changed by the weight of the edge. Given a fixed starting vertex and initial energy, Alice wins the game if the energy of the token remains nonnegative at every moment. If the energy goes below zero at some point, then Bob wins. The problem of determining the winner in an energy game lies in NP ∩ coNP. It is a long standing open problem whether a polynomial time algorithm for this problem exists. We devise new algorithms for three special cases of the problem. The first two results focus on the single-player version, where either Alice or Bob controls the whole game graph. We develop an˜() time algorithm for a game graph controlled by Alice, by providing a reduction to the All-Pairs Nonnegative Prefix Paths problem (APNP), where is the maximum absolute value of any edge weight and is the best exponent for matrix multiplication. Thus we study the APNP problem separately, for which we develop an˜() time algorithm. For both problems, we improve over the state of the art of˜() for small. For the APNP problem, we also provide a conditional lower bound which states that there is no (3−) time algorithm for any > 0, unless the APSP Hypothesis fails. For a game graph controlled by Bob, we obtain a near-linear time algorithm. Regarding our third result, we present a variant of the value iteration algorithm, and we prove that it gives an () time algorithm for game graphs without negative cycles, which improves a previous upper bound. The all-Bob algorithm is randomized, all other algorithms are deterministic.

Turn-based stochastic games and its important subclass Markov decision processes (MDPs) provide models for systems with both probabilistic and nondeterministic behaviors. We consider turn-based stochastic games with two classical quantitative objectives: discounted-sum and long-run average objectives. The game models and the quantitative objectives are widely used in probabilistic verification, planning, optimal inventory control, network protocol and performance analysis. Games and

Recent work on approximate linear programming (ALP) techniques for first-order Markov Decision Processes (FOMDPs) represents the value function linearly w.r.t. a set of first-order basis functions and uses linear programming techniques to determine suitable weights. This approach offers the advantage that it does not require simplification of the first-order value function, and allows one to solve FOMDPs independent of a specific domain instantiation. In this paper, we address several questions to enhance the applicability of this work: (1) Can we extend the first-order ALP framework to approximate policy iteration and if so, how do these two algorithms compare? (2) Can we automatically generate basis functions and evaluate their impact on value function quality? (3) How can we decompose intractable problems with universally quantified rewards into tractable subproblems? We propose answers to these questions along with a number of novel optimizations and provide a comparative empirical evaluation on problems from the ICAPS 2004 Probabilistic Planning Competition.

I am most grateful to my supervisors Prof. Elizabeth Jewkes and Prof. Qi-Ming He for their guidance, support and inspiration throughout my Ph.D. studies. Thank you Prof. Jewkes for your encouragement and support during my Masters degree as well. I would like to thank the members of my examination committee, Prof. Armann Ingolfsson, Prof. Steve Drekic, Prof. Samir Elhedhli and Prof. Fatma Gzara. I am very grateful for their insightful comments and helpful suggestions on my thesis that improved this research. I wish to express my gratitude to all my friends and colleagues at the Department of Management Sciences who were always there when I needed them. And for making this an enjoyable experience. Special thanks to Bissan, Hossa, Mina and Tiffany. I am greatly indebted to my husband Islam Saleh who has provided me with unconditional love and support throughout my long journey. Without him, I wouldn't have achieved this success. Also, I am indebted for my sweet little kids Noor and Dania for their patience and understanding when I was busy. Finally, I am forever indebted to my parents, brothers and sister for their constant care, support, and encouragements. v Contents List of Figures ix List of Tables x

I am most grateful to my supervisors Prof. Elizabeth Jewkes and Prof. Qi-Ming He for their guidance, support and inspiration throughout my Ph.D. studies. Thank you Prof. Jewkes for your encouragement and support during my Masters degree as well. I would like to thank the members of my examination committee, Prof. Armann Ingolfsson, Prof. Steve Drekic, Prof. Samir Elhedhli and Prof. Fatma Gzara. I am very grateful for their insightful comments and helpful suggestions on my thesis that improved this research. I wish to express my gratitude to all my friends and colleagues at the Department of Management Sciences who were always there when I needed them. And for making this an enjoyable experience. Special thanks to Bissan, Hossa, Mina and Tiffany. I am greatly indebted to my husband Islam Saleh who has provided me with unconditional love and support throughout my long journey. Without him, I wouldn't have achieved this success. Also, I am indebted for my sweet little kids Noor and Dania for their patience and understanding when I was busy. Finally, I am forever indebted to my parents, brothers and sister for their constant care, support, and encouragements. v Contents List of Figures ix List of Tables x

We consider the optimal production and inventory control of an assemble-to-order system with m components, one end-product, and n customer classes. A control policy specifies when to produce each component and, whenever an order is placed, whether or not to satisfy it from on-hand inventory. We formulate the problem as a Markov decision process and characterize the structure of an optimal policy. We show that a base-stock production policy is optimal, but the base-stock level for each component is dynamic and depends on the inventory level of all other components (more specifically, it is nondecreasing). We show that the optimal inventory allocation for each component is a rationing policy with different rationing levels for different demand classes. The rationing levels for each component are dynamic and also nondecreasing in the inventory level of all other components. We compare the performance of the optimal policy to heuristic policies, including the commonly used base-stock po...

We consider a production-inventory system with two customer classes, one patient and one impatient. Orders from the patient class can be backordered if needed while orders from the impatient class must be rejected if they cannot be fulfilled from on-hand inventory. Orders backordered incur a backorder cost while orders rejected incur a lost sales cost. The objective is to minimize the sum of inventory holding cost and the costs of backorders and lost sales. We formulate the problem as a Markov decision process and use this formulation to characterize the structure of the optimal policy. We show that the optimal policy can be described by two threshold functions that depend on the level of backorders from the patient class. These threshold functions specify (1) when it is optimal to produce, (2) how to allocate units produced to either increase inventory or reduce backorder, and (3) when to fulfill orders from on-hand inventory and when to backorder (in the case of the patient class) and when to reject them (in the case of the impatient class). We show that the priority in inventory allocation among the two classes is not static and instead depends on the backorder level from the class of patient customers. In particular, it is possible to start out fulfilling orders from the impatient class and backordering orders from the patient class and then to switch to fulfilling orders from the patient class and rejecting orders from the impatient class. In addition to characterizing the structure of the optimal policy, we also describe an effective heuristic that retains the essential features of the optimal policy but is significantly simpler to implement. This heuristic performs nearly as well as the optimal policy and significantly outperforms other plausible heuristics.

Calls of two classes arrive at a call center according to two independent Poisson processes. The center has two dedicated stations, one for each class, and one shared station. All three stations consist of parallel servers and no waiting room. Calls of each type demand exponential service times with different service rates and generate different rewards. Moreover, the service rates are different in the shared and dedicated stations. We assume non-preemptive service. Our objective is to derive the structure of dynamic admission policies that maximize the total expected discounted revenue over an infinite horizon as well as the long-run average revenue. We show that it is optimal to serve a customer in her dedicated station whenever it is possible. For the shared station, we derive a sufficient condition for each class under which it is always optimal to accept customers of that class to the shared station if the dedicated station is full and the shared station has available servers. Furthermore, the optimal admission policy at the shared station can be characterized as a monotonic threshold policy.

Reinforcement Learning (RL) is being increasingly applied to optimize complex functions that may have a stochastic component. RL is extended to multi-agent systems to find policies to optimize systems that require agents to coordinate or to compete under the umbrella of Multi-Agent RL (MARL). A crucial factor in the success of RL is that the optimization problem is represented as the expected sum of rewards, which allows the use of backward induction for the solution. However, many real-world problems require a joint objective that is non-linear and dynamic programming cannot be applied directly. For example, in a resource allocation problem, one of the objective is to maximize long-term fairness among the users. This paper addresses and formalizes the problem of joint objective optimization, where not only the sum of rewards of each agent but a function of the sum of rewards of each agent needs to be optimized. The proposed algorithms at the centralized controller aims to learn the...

Probabilistic model checking is a formal method for verification of the quantitative and qualitative properties of computer systems with stochastic behaviors. Markov Decision Processes (MDPs) are well-known formalisms for modeling this class of systems. Bounded reachability probabilities are an important class of properties that are computed in probabilistic model checking. Iterative numerical computations are used for this class of properties. A significant drawback of the standard iterative methods is the redundant computations that do not affect the final results of the computations, but increase the running time of the computations. The study proposes two new approaches to avoid redundant computations for bounded reachability analysis. The general idea of these approaches is to identify and avoid useless numerical computations in iterative methods for computing bounded reachability probabilities.

Predictive linguistics is a growing field at the interface of language sciences, cognitive sciences, and artificial intelligence, focusing on how humans and machines use predictive processes to process and produce language. This discipline is distinguished by its proactive approach, emphasizing the internal mechanisms that allow us to anticipate future linguistic structures. It explores the underlying processes at all levels of language-morphological, syntactic, semantic, pragmatic, etc.and aims to understand how these predictions influence the production and comprehension of discourse.
The main goal of predictive linguistics is to model cognitive mechanisms to better understand how the brain processes language, but also to develop models capable of generating, predicting, and understanding language in a manner similar to human cognition. By focusing on inner speech and the cognitive mechanisms that allow us to predict linguistic structures at different levels, predictive linguistics aims to create language models capable of reproducing these predictive capabilities. In doing so, it could revolutionize natural language processing and pave the way for artificial general intelligence, capable of linguistic awareness similar to that of humans.

This paper presents a novel algorithm for learning in a class of stochastic Markov decision processes (MDPs) with continuous state and action spaces that trades speed for accuracy. A transform of the stochastic MDP into a deterministic one is presented which captures the essence of the original dynamics, in a sense made precise. In this transformed MDP, the calculation of values is greatly simplified. The online algorithm estimates the model of the transformed MDP and simultaneously does policy search against it. Bounds on the error of this approximation are proven, and experimental results in a bicycle riding domain are presented. The algorithm learns near optimal policies in orders of magnitude fewer interactions with the stochastic MDP, using less domain knowledge. All code used in the experiments is available on the project's web site.

Non-stationary domains, where unforeseen changes happen, present a challenge for agents to find an optimal policy for a sequential decision making problem. This work investigates a solution to this problem that combines Markov Decision Processes (MDP) and Reinforcement Learning (RL) with Answer Set Programming (ASP) in a method we call ASP(RL). In this method, Answer Set Programming is used to find the possible trajectories of an MDP, from where Reinforcement Learning is applied to learn the optimal policy of the problem. Results show that ASP(RL) is capable of efficiently finding the optimal solution of an MDP representing non-stationary domains.

On the basis of a database of more than 80 thousand records on total retails and production costs of the pharmaceutical industry worldwide we consider four classes of drugs. We evaluate the expected profits of an investment in a new drug in the four classes of pharmaceutical products by considering the standard NPV evaluation. We compare these outcomes with the evaluation of the expected profits of the four new drugs obtained by the real option approach. Interestingly enough quite different outcomes are obtained. These results loom on the capacity of standard methods to give a reliable evaluation of real investment projects that are analogous to compound options.

Climatic changes will affect the occurrence probability of extreme windstorms. Consequently, management of uneven-aged forests can only be optimized correctly if changes in climatic conditions are considered. This article determines the optimal management regime of uneven-aged forests in the presence of climatic changes affecting windstorm occurrence probabilities, and takes into account the risk preferences of the forest owner. This study analyzes optimal harvesting of uneven-aged stands by applying a Markov Decision Processes (MDP) framework and an economic description of uneven-aged forestry. Two management types are considered: the exact uneven-aged forest management model in which the forest owner jointly manages all the different stand plots, and the independent uneven-aged forest management model that assumes that the forest owner separately and independently manages each plot of the forest. The MDP framework is applied to a non-industrial private forest owner located in Nort...

Confronted by significant impacts to ecosystems world‐wide, decision makers face the challenge of maintaining both biodiversity and the provision of ecosystem services (ES). However, the objectives of managing biodiversity and supplying ES may not always be in concert, resulting in the need for trade‐offs. Understanding these potential trade‐offs is crucial for identifying circumstances under which conservation strategies designed to maximise either biodiversity or ES will result in win‐win or win‐lose outcomes. One important factor that may influence these outcomes are species interactions and the structure of the networks in which they are embedded. We combine optimisation and network theory to investigate the difference in species prioritisation and management outcomes when targeting biodiversity or ES, by considering trophic interactions between species. We analyse 360 simulated ecosystem networks with different ecosystem structures, including the trophic level of the species pr...

Classical stochastic Markov Decision Processes (MDPs) and possibilistic MDPs (-MDPs) aim at solving the same kind of problems, involving sequential decision making under uncertainty. The underlying uncertainty model (probabilistic / possibilistic) and preference model (reward / satisfaction degree) change, but the algorithms, based on dynamic programming, are similar. So, a question maybe raised about when to prefer one model to another, and for which reasons. The answer may seem obvious when the uncertainty i s o f a n objective nature (symmetry of the problem, frequentist information) and when the problem is faced repetitively and rewards accumulate. It is less clear when uncertainty and preferences are qualitative, purely subjective and when the problem is faced only once. In this paper we carry out an empirical comparison of both types of algorithms (stochastic and possibilistic), in terms of \quality" of the solutions, and time needed to compute them.

The output of the system is a sequence of actions in some applications. There is no such measure as the best action in any in-between state; an action is excellent if it is part of a good policy. A single action is not important; the policy is important that is the sequence of correct actions to reach the goal. To be able to generate a policy the machine learning programs should able to assess the quality of policies and learn from past good action sequences. Learning is the basic capacity of intelligent agents. An agent changes its behavior based on its previous experiences through learning. An intelligent agent must be formalized by knowledge and be able to act on this knowledge. In many single-agent systems for learning the policy of an agent in uncertain environments, the reinforcement learning techniques have been applied successfully. Many existing single-agent models for sequential decision making are derived from a general model and are distinguished by assumptions. Q-learni...

In large-scale persistent missions, the vehicle capabilities and health often degrade over time. This paper presents a Health Aware Planning (HAP) Framework for longduration complex UAV missions by establishing close feedback between the high-level planning based on Markov Decision Processes (MDP) and the execution level learning-focused adaptive controllers. This feedback enables the HAP framework to plan by anticipating the failures and reassessing vehicle capabilities after the failures. This proactive behavior allows for efficient replanning to account for changing capabilities. Simulations for a 4 UAV target tracking scenario is presented to demonstrate the effectiveness of the proactive replanning capability of the presented HAP framework.

This paper deals with approximate value iteration (AVI) algorithms applied to discounted dynamic programming (DP) problems. For a fixed control policy, the span semi-norm of the so-called Bellman residual is shown to be convex in the Banach space of candidate solutions to the DP problem. This fact motivates the introduction of an AVI algorithm with local search that seeks to minimize the span semi-norm of the Bellman residual in a convex value function approximation space. The novelty here is that the optimality of a point in the approximation architecture is characterized by means of convex optimization concepts and necessary and sufficient conditions to local optimality are derived. The procedure employs the classical AVI algorithm direction (Bellman residual) combined with a set of independent search directions, to improve the convergence rate. It has guaranteed convergence and satisfies, at least, the necessary optimality conditions over a prescribed set of directions. To illustrate the method, examples are presented that deal with a class of problems from the literature and a large state space queueing problem setting.

This note provides upper bounds on the number of operations required to compute by value iterations a nearly optimal policy for an infinite-horizon discounted Markov decision process with a finite number of states and actions. For a given discount factor, magnitude of the reward function, and desired closeness to optimality, these upper bounds are strongly polynomial in the number of state-action pairs, and one of the provided upper bounds has the property that it is a non-decreasing function of the value of the discount factor.

This paper establishes new links between stochastic and discrete optimization. We consider the following three problems for discrete time Markov Decision Processes with finite states and action sets: (i) find an optimal deterministic policy for a discounted problem with constraints, (ii) find an optimal stationary policy for a weighted discounted problem with constraints, (iii) find an optimal deterministic policy for a weighted discounted problem with constraints. We formulate mathematical programs for problems (i)–(iii) and show that the Hamiltonian Cycle Problem is a special case of each of these problems. Therefore problems (i)–(iii) are NP-hard. We also provide new mathematical programming formulations for the Hamiltonian Cycle and Traveling Salesman Problems.

This paper deals with constrained optimization of Markov Decision Processes. Both objective function and constraints are sums of standard discounted rewards, but each with a different discount factor Such models arise, e.g., in production and in applications involving multiple time scales. We prove that it a feasible policy exists, then there exists an optimal policy which is (i) stationary (nonrandomized) from some step onward, (ii) randomized, Markov before this step, but the total number of actions which are added by randomization is bounded by the number of constraints. Optimality of such policies for multi-criteria problems is also established. These new policies have the pleasing aesthetic property that the amount of randomization they require over any trajectory is restricted by the number of constraints. This result is new even for constrained optimization with a single discount factor, where the optimality of randomized stationary policies is known. However, a randomized st...

We investigate logics and equivalence relations that capture the qualitative behavior of Markov Decision Processes (MDPs). We present Qualitative Randomized Ctl (Qrctl): formulas of this logic can express the fact that certain temporal properties hold over all paths, or with probability 0 or 1, but they do not distinguish among intermediate probability values. We present a symbolic, polynomial time model-checking algorithm for Qrctl on MDPs. The logic Qrctl induces an equivalence relation over states of an MDP that we call qualitative equivalence: informally, two states are qualitatively equivalent if the sets of formulas that hold with probability 0 or 1 at the two states are the same. We show that for finite alternating MDPs, where nondeterministic and probabilistic choices occur in different states, qualitative equivalence coincides with alternating bisimulation, and can thus be computed via efficient partition-refinement algorithms. On the other hand, in nonalternating MDPs the equivalence relations cannot be computed via partition-refinement algorithms, but rather, they require non-local computation. Finally, we consider Qrctl * , that extends Qrctl with nested temporal operators in the same manner in which Ctl * extends Ctl. We show that Qrctl and Qrctl * induce the same qualitative equivalence on alternating MDPs, while on non-alternating MDPs, the equivalence arising from Qrctl * can be strictly finer. We also provide a full characterization of the relation between qualitative equivalence, bisimulation, and alternating bisimulation, according to whether the MDPs are finite, and to whether their transition relations are finitely-branching.

Turn-based stochastic games and its important subclass Markov decision processes (MDPs) provide models for systems with both probabilistic and nondeterministic behaviors. We consider turn-based stochastic games with two classical quantitative objectives: discounted-sum and long-run average objectives. The game models and the quantitative objectives are widely used in probabilistic verification, planning, optimal inventory control, network protocol and performance analysis. Games and

This paper proposes a new approach to modelling and controlling Internet end-to-end loss behaviours. Rather than select the model structure from the loss observations as being done previously, we construct a new loss model based on the TCP congestion control mechanisms. Thus, the model can explicitly reflect the correlation between end-to-end loss observations and network flow level activities. Besides simulation, the model has been tested in both wired and wireless Internet environments. The result shows that, unless the losses due to the transmission errors are excessive e.g. in some lossy wireless channels, the model can correctly capture end-to-end loss behaviours not only in terms of average rates but also in terms of loss patterns i.e. loss and good runlengths. This implies a good connection between the model structure and network flow level activities, which makes the model attractive for assisting network traffic control.

The use of multipath routing in overlay networks is a promising solution to improve performance and availability of Internet applications, without the replacement of the existing TCP/IP infrastructure. In this paper, we propose an approach to distribute data over multiple overlay paths that is able to improve Quality of Service (QoS) metrics, such as the data transfer time, loss, and throughput. By using the Imbedded Markov Chain technique, we demonstrate that the system under analysis, observed at specific instants, possesses the Markov property. We therefore cast the data distribution problem into the Markov Decision Process (MDP) framework, and design a computationally efficient algorithm named Online Policy Iteration (OPI), to solve the optimization problem on the fly. The proposed approach is applied to the problem of multipath data distribution in various wired/wireless network scenarios, with the objective of minimizing the data transfer time as well as the delay and losses. Through both intensive ns-2 simulations with data collected from real heterogeneous networks and experiments over real networks, we show the superior performance of the proposed traffic control mechanism in comparison with two classical schemes, that are Weighted Round Robin and Join the Shortest Queue.

We consider the problem of finding good fi nite-horizon policies for POMDPs under the expected reward metric. The policies con sidered are free finite-memory policies with limited memory; a policy is a mapping from the space of observation-memory pairs to the space of action-memory pairs (the policy up dates the memory as it goes), and the num ber of possible memory states is a parameter of the input to the policy-finding algorithms. The algorithms considered here are prelimi nary implementations of three search heuris tics: local search, simulated annealing, and genetic algorithms. We compare their out comes to each other and to the optimal poli cies for each instance. We compare run times of each policy and of a dynamic programming algorithm for POMDPs developed by Hansen that iteratively improves a fi nite-state con troller-the previous state of the art for finite memory policies. The value of the best policy can only improve as the amount of memory increases, up to the amount needed for an optimal finite-memory policy. Our most surprising finding is that more memory helps in another way: given more memory than is needed for an optimal policy, the algo rithms are more likely to converge to optimal valued policies.

Solving partially observable Markov decision processes (POMDPs) is highly intractable in gen eral, at least in part because the optimal policy may be infi nitely large. In this paper, we ex plore the problem of fi nding the optimal policy from a restricted set of policies, represented as fi nite state automata of a given size. This prob lem is also intractable, but we show that the com plexity can be greatly reduced when the POMDP and/or policy are further constrained. We demon strate good empirical results with a branch-and bound method for fi nding globally optimal deter ministic policies, and a gradient-ascent method for fi nding locally optimal stochastic policies.

We investigate the use of temporally abstract actions, or macro-actions, in the solution of Markov decision processes. Unlike current mod els that combine both primitive actions and macro-actions and leave the state space un changed, we propose a hierarchical model (using an abstract MDP) that works with macro-actions only, and that signifi cantly reduces the size of the state space. This is achieved by treating macro actions as local policies that act in certain regions of state space, and by restricting states in the ab stract MDP to those at the boundaries of regions. The abstract MDP approximates the original and can be solved more efficiently. We discuss sev eral ways in which macro-actions can be gen erated to ensure good solution quality. Finally, we consider ways in which macro-actions can be reused to solve multiple, related MDPs; and we show that this can justify the computational over head of macro-action generation. 1 Introduction Markov decision processes (MDPs) [11, 22] have proven tremendously useful as models of stochastic planning and decision problems. However, traditional dynamic pro gramm ing remains computationally intractable for practi cal problems, requiring time polynomial in the size of the state and action spaces, but where these spaces are gener ally too large to be explicitly enumerated. Considerable research has been directed toward the solution of Markov decision processes (MDPs) with large state and action spaces. These include function approximation [2], reach ability analyses [5] and aggregation techniques [7, 3, 4]. Despite these advances, little attention has been paid to the reuse of policies or value functions generated for one MDP in the solution of a related MDP. While such reasoning is

The well-known Kullback-Leibler divergence of a random ÿeld from its factorization quantiÿes spatial interdependences of the corresponding stochastic elements. We introduce a generalized measure called 'stochastic interaction' that captures also temporal interdependences. Maximization of stochastic interaction in the setting of Markov chains is shown analytically and by simulations to result in an almost deterministic global dynamics, but almost unpredictable single unit activity.

Given a 0-1 infinite matrix A and its countable Markov shift ΣA, one of the authors and M. Laca have introduced a kind of generalized countable Markov shift XA = ΣA ∪ YA, where YA is a special set of finite admissible words. For some of the most studied countable Markov shifts ΣA, XA is a compactification of ΣA, and always it is at least locally compact. We developed the thermodynamic formalism on the space XA, exploring the connections with standard results on ΣA. New phenomena appear, such as new conformal measures and a lengthtype phase transition: the eigenmeasure lives on ΣA at high temperature and lives on YA at low temperature. Using a pressure-point definition proposed by M. Denker and M. Yuri for iterated function systems, we proved that the Gurevich pressure is a natural definition for the pressure function in the generalized setting. For the gauge action, the Gurevich entropy is a critical temperature for the existence of new conformal measures (KMS states) living on YA. We exhibit examples with infinitely (even uncountable) many new extremal conformal measures, undetectable in the usual formalism. We prove that conformal measures always exist at low temperatures when the potential is coercive enough. We characterized a basis of the topology of XA to study the weak * convergence of measures on XA, and we show some cases where the conformal measure living on YA converges to a conformal one living on ΣA. We prove the equivalence among several notions of conformality for locally compact Hausdorff second countable spaces, including quasi-invariant measures for generalized Renault-Deaconu groupoids. Contents 6.1. Renewal shift 57 6.2. Pair renewal shift 61 6.3. Prime renewal shift: infinitely many extremal conformal measures 65 6.4. Infinite entropy and uncountably many extremal conformal measures 65 6.5. Infinite emitters, Y A-families and extremal conformal measures 68 7. Eigenmeasures 69 7.1. An example of length-type phase transition 72 8. Concluding remarks 77 Acknowledgements 78 9. Appendix 79 9.1. Auxiliary results 79 9.2. Proof of the Theorem 65 82 References 85