A CONVEX ANALYTIC APPROACH TO RISK-AWARE MARKOV DECISION PROCESSES

作者：Haskell William B^*; Jain Rahul

来源：SIAM Journal on Control and Optimization, 2015, 53(3): 1569-1598.

DOI：10.1137/140969221

• Markov decision processes
• stochastic optimization
• risk measures
• conditional value-at-risk
• stochastic dominance constraints
• convex analytic approach

In classical Markov decision process (MDP) theory, we search for a policy that, say, minimizes the expected infinite horizon discounted cost. Expectation is, of course, a risk neutral measure, which does not suffice in many applications, particularly in finance. We replace the expectation with a general risk functional, and call such models risk-aware MDP models. We consider minimization of such risk functionals in two cases, the expected utility framework, and conditional value-at-risk, a popular coherent risk measure. Later, we consider risk-aware MDPs wherein the risk is expressed in the constraints. This includes stochastic dominance constraints, and the classical chance-constrained optimization problems. In each case, we develop a convex analytic approach to solve such risk-aware MDPs. In most cases, we show that the problem can be formulated as an infinite-dimensional linear program (LP) in occupation measures when we augment the state space. We provide a discretization method and finite approximations for solving the resulting LPs. A striking result is that the chance-constrained MDP problem can be posed as an LP via the convex analytic method.

• 出版日期2015