An Equivalence Between Adaptive Dynamic Programming With a Critic and Backpropagation Through Time

作者：Fairbank Michael^*; Alonso Eduardo; Prokhorov Danil

来源：IEEE Transactions on Neural Networks and Learning Systems, 2013, 24(12): 2088-2100.

DOI：10.1109/TNNLS.2013.2271778

• Adaptive dynamic programming (ADP)
• backpropagation through time
• dual heuristic programming (DHP)
• neural networks
• value-gradient learning

We consider the adaptive dynamic programming technique called Dual Heuristic Programming (DHP), which is designed to learn a critic function, when using learned model functions of the environment. DHP is designed for optimizing control problems in large and continuous state spaces. We extend DHP into a new algorithm that we call Value-Gradient Learning, VGL (lambda), and prove equivalence of an instance of the new algorithm to Backpropagation Through Time for Control with a greedy policy. Not only does this equivalence provide a link between these two different approaches, but it also enables our variant of DHP to have guaranteed convergence, under certain smoothness conditions and a greedy policy, when using a general smooth nonlinear function approximator for the critic. We consider several experimental scenarios including some that prove divergence of DHP under a greedy policy, which contrasts against our proven-convergent algorithm.

• 出版日期2013-12