Fuzzy optimality relation for perceptive MDPs—the average case

2021

This paper concerns discounted Markov decision processes with a fuzzy reward function triangular in shape. Starting with a usual and non-fuzzy Markov control model (Hernández-Lerma, 1989) with compact action sets and reward R, a control model is induced only substituting R in the usual model for a suitable triangular fuzzy function R̃ which models, in a fuzzy sense, the fact that the reward R is “approximately” received. This way, for this induced model a discounted optimal control problem is considered, taking into account both a finite and an infinite horizons, and fuzzy objective functions. In order to obtain the optimal solution, the partial order on the α-cuts of fuzzy numbers is used, and the optimal solution for fuzzy Markov decision processes is found from the optimal solution of the corresponding usual Markov decision processes. In the end of the paper, several examples are given to illustrate the theory developed: a model of inventory system, and two others more in an econ...

Kybernetika, 2023

The article presents an extension of the theory of standard Markov decision processes on discrete spaces and with the average cost as the objective function which permits to take into account a fuzzy average cost of a trapezoidal type. In this context, the fuzzy optimal control problem is considered with respect to two cases: the max-order of the fuzzy numbers and the average ranking order of the trapezoidal fuzzy numbers. Each of these cases extends the standard optimal control problem, and for each of them the optimal solution is related to a suitable standard optimal control problem, and it is obtained that (i) the optimal policy coincides with the optimal policy of this suitable standard control problem, and (ii) the fuzzy optimal value function is of a trapezoidal shape. Two models: a queueing system and a machine replacement problem are provided in order to examplify the theory given.

In this paper, we consider the model that the information on the rewards in vector-valued Markov decision processes includes imprecision or ambiguity. The fuzzy reward model is analyzed as follows: The fuzzy reward is represented by the fuzzy set on the multi-dimensional Euclidian space R and the infinite horizon fuzzy expected discounted reward(FEDR) from any stationary policy is characterized as a unique fixed point of the corresponding contractive operator. Also, we fined a Pareto optimal policy which maximizes the infinite horizon FEDR over all stationary policies under the pseudo order induced by a convex cone R . As a numerical example, the machine maintenance problem is considered.

Fuzzy Sets and Systems, 1994

This paper constructs a Markov fuzzy process, which represents transitions of grades of fuzzy sets, with a transition possibility measure and a general state space. We analyse Snell's optimal stopping problem for the process and we apply the results to solve fuzzy dynamic programming with optimal stopping times and with general state spaces and action spaces under fuzzy transitions.

European Journal of Operational Research, 1996

We will define a new multi-stage decision process, which is termed Markov-type fuzzy decision process. By the general framework in the decision process, the optimization problem of the discounted reward is discussed under a partial order of convex fuzzy numbers.

Fuzzy Optimization and Decision Making, 2002

Decision-making in an environment of uncertainty and imprecision for real-world problems is a complex task. In this paper it is introduced general finite state fuzzy Markov chains that have a finite convergence to a stationary (may be periodic) solution. The Cesaro average and the α-potential for fuzzy Markov chains are defined, then it is shown that the relationship between them corresponds to the Blackwell formula in the classical theory of Markov decision processes. Furthermore, it is pointed out that recurrency does not necessarily imply ergodicity. However, if a fuzzy Markov chain is ergodic, then the rows of its ergodic projection equal the greatest eigen fuzzy set of the transition matrix. Then, the fuzzy Markov chain is shown to be a robust system with respect to small perturbations of the transition matrix, which is not the case for the classical probabilistic Markov chains. Fuzzy Markov decision processes are finally introduced and discussed.

In this paper, we shall develop a fuzzy treatment for uncertain Markov decision processes which allow for a fluctuating transition matrix at each step in time. The decision model with uncertain transition matrices is described by the use of fuzzy sets in which we find a Pareto optimal policy maximizing the infinite horizon fuzzy expected discounted reward over all stationary policies under some partial order. The Pareto optimal policies are characterized by maximal solutions of an optimal equation including efficient set-functions. As a numerical example, the machine maintenance problem is considered.

Top, 2006

This paper is a survey of recent results on continuous-time Markov decision processes (MDPs) withunbounded transition rates, and reward rates that may beunbounded from above and from below. These results pertain to discounted and average reward optimality criteria, which are the most commonly used criteria, and also to more selective concepts, such as bias optimality and sensitive discount criteria. For concreteness, we consider only MDPs with a countable state space, but we indicate how the results can be extended to more general MDPs or to Markov games.

The Annals of Applied Probability, 2006

This paper is devoted to studying the average optimality in continuous-time Markov decision processes with fairly general state and action spaces. The criterion to be maximized is expected average rewards. The transition rates of underlying continuous-time jump Markov processes are allowed to be unbounded, and the reward rates may have neither upper nor lower bounds. We first provide two optimality inequalities with opposed directions, and also give suitable conditions under which the existence of solutions to the two optimality inequalities is ensured. Then, from the two optimality inequalities we prove the existence of optimal (deterministic) stationary policies by using the Dynkin formula. Moreover, we present a "semimartingale characterization" of an optimal stationary policy. Finally, we use a generalized Potlach process with control to illustrate the difference between our conditions and those in the previous literature, and then further apply our results to average optimal control problems of generalized birth-death systems, upwardly skip-free processes and two queueing systems. The approach developed in this paper is slightly different from the "optimality inequality approach" widely used in the previous literature. . This reprint differs from the original in pagination and typographic detail. 1 2 X. GUO AND U. RIEDER specified by four primitive data: a state space S; an action space A with subsets A(x) of admissible actions, which may depend on the current state x ∈ S; transition rates q(·|x, a); and reward (or cost) rates r(x, a). Using these terms, we now briefly describe some existing works on the expected average criterion. When the state space is finite, a bounded solution to the average optimality equation (AOE) and methods for computing optimal stationary policies have been investigated in . Since then, most work has focused on the case of a denumerable state space; for instance, see for bounded transition and reward rates, for bounded transition rates but unbounded reward rates, [16, 35] for unbounded transition rates but bounded reward rates and [12, 13, 17] for unbounded transition and reward rates. For the case of an arbitrary state space, to the best of our knowledge, only Doshi [5] and Hernández-Lerma [19] have addressed this issue. They ensured the existence of optimal stationary policies. However, the treatments in [5] and [19] are restricted to uniformly bounded reward rates and nonnegative cost rates, respectively, and the AOE plays a key role in the proof of the existence of average optimal policies. Moreover, to establish the AOE, Doshi [5] needed the hypothesis that all admissible action sets are finite and the relative difference of the optimal discounted value function is equicontinuous, whereas in [19] the assumption about the existence of a solution to the AOE is imposed. On the other hand, it is worth mentioning that some of the conditions in are imposed on the family of weak infinitesimal operators deduced from all admissible policies, instead of the primitive data. In this paper we study the much more general case. That is, the reward rates may have neither upper nor lower bounds, all of the state and action spaces are fairly general and the transition rates are allowed to be unbounded. We first provide two optimality inequalities rather than one for the "optimality inequality approach" used in , for instance. Under suitable assumptions we not only prove the existence of solutions to the two optimality inequalities, but also ensure the existence of optimal stationary policies by using the two inequalities and the Dynkin formula. Also, to verify our assumptions, we further give sufficient conditions which are imposed on the primitive data. Moreover, we present a semimartingale characterization of an optimal stationary policy. Finally, we use controlled generalized Potlach processes to show that all conditions in this paper are satisfied, whereas the earlier conditions fail to hold. Then we further apply our results to average optimal control problems of generalized birth-death systems and upwardly skip-free processes [1], a pair of controlled queues in tandem , and M/M/N/0 queue systems . It should be noted that, on the one hand, the optimality inequality approach used in the previous literature (see, e.g., for continuous-time MDPs and [20, 21, 31, 34] for discrete-time MDPs) is not applied to our case, because in our model the reward rates may have neither upper nor lower bounds. On the other hand, we not only CONTINUOUS-TIME MARKOV DECISION PROCESSES 3

Fuzzy Sets and Systems, 2006

This article was originally published in a journal published by Elsevier, and the attached copy is provided by Elsevier for the author's benefit and for the benefit of the author's institution, for non-commercial research and educational use including without limitation use in instruction at your institution, sending it to specific colleagues that you know, and providing a copy to your institution's administrator.