Neuroevolution strategies for episodic reinforcement learning

作者：Heidrich Meisner Verena^*; Igel Christian

来源：Journal of Algorithms, 2009, 64(4): 152-168.

DOI：10.1016/j.jalgor.2009.04.002

• Reinforcement learning
• Evolution strategy
• Covariance matrix adaptation
• Partially observable Markov decision process
• Direct policy search

Because of their convincing performance, there is a growing interest ill using evolutionary algorithms for reinforcement learning. We propose learning of neural network policies by. the covariance matrix adaptation evolution strategy (CMA-ES), a randomized variable-metric search algorithm for continuous optimization. We argue that this approach, which we refer to as CMA Neuroevolution Strategy (CMA-NeuroES), is ideally suited reinforcement learning, in particular because it is based on ranking policies (and therefore robust against noise), efficiently detects correlations between parameters, and infers a search direction from scalar reinforcement signals. We evaluate the CMA-NeuroES I on five different (Markovian and non-Markovian) variants of the common pole balancing problem. The results are compared to those described in a recent study covering several RL algorithms, and the CMA-NeuroES shows the overall best performance.

• 出版日期2009-10