Steering approaches to Pareto-optimal multiobjective reinforcement learning

作者：Vamplew Peter^*; Issabekov Rustam; Dazeley Richard; Foale Cameron; Berry Adam; Moore Tim; Creighton Douglas

来源：Neurocomputing, 2017, 263: 26-38.

DOI：10.1016/j.neucom.2016.08.152

• Multiobjective reinforcement learning
• Non-stationary policies
• Geometric steering
• Interactive reinforcement learning
• Pareto optimality

For reinforcement learning tasks with multiple objectives, it may be advantageous to learn stochastic or non-stationary policies. This paper investigates two novel algorithms for learning non-stationary policies which produce Pareto-optimal behaviour (w-steering and Q-steering), by extending prior work based on the concept of geometric steering. Empirical results demonstrate that both new algorithms offer substantial performance improvements over stationary deterministic policies, while Q-steering significantly outperforms w-steering when the agent has no information about recurrent states within the environment. It is further demonstrated that Q-steering can be used interactively by providing a human decision-maker with a visualisation of the Pareto front and allowing them to adjust the agent's target point during learning. To demonstrate broader applicability, the use of Q-steering in combination with function approximation is also illustrated on a task involving control of local battery storage for a residential solar power system.

• 出版日期2017-11-8
• 单位迪肯大学; CSIRO