An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes

作者：Bhatnagar Shalabh^*

来源：Systems & Control Letters, 2010, 59(12): 760-766.

DOI：10.1016/j.sysconle.2010.08.013

• Constrained Markov decision processes
• Infinite horizon discounted cost criterion
• Function approximation
• Actor-critic algorithm
• Simultaneous perturbation stochastic approximation

We develop in this article the first actor-critic reinforcement learning algorithm with function approximation for a problem of control under multiple inequality constraints. We consider the infinite horizon discounted cost framework in which both the objective and the constraint functions are suitable expected policy-dependent discounted sums of certain sample path functions. We apply the Lagrange multiplier method to handle the inequality constraints. Our algorithm makes use of multi-timescale stochastic approximation and incorporates a temporal difference (TD) critic and an actor that makes a gradient search in the space of policy parameters using efficient simultaneous perturbation stochastic approximation (SPSA) gradient estimates. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal policy.

• 出版日期2010-12