Actor-Critic Learning Based on Adaptive Importance Sampling

作者：Cheng Yuhu^*; Feng Huanting; Wang Xuesong

来源：Chinese Journal of Electronics, 2010, 19(4): 583-588.

• Actor-Critic learning
• Policy gradient
• Adaptive importance sampling
• Least squares temporal difference
• Importance-weights cross-validation

For on-policy Actor-critic (AC) reinforcement learning, sampling is a time-consuming and expensive work In order to efficiently reuse previously collected samples and to reduce large estimation variance, a kind of off-policy AC learning algorithm based on an Adaptive importance sampling (AIS) technique is proposed The Critic estimates the value-function using the least squares temporal difference with eligibility trace and the AIS technique In order to control the trade-off between bias and variance of the estimation of policy gradient, a flattening factor is Introduced to the importance weight in the AIS The value of the flattening factor can be determined by an importance-weight cross-validation method automatically from samples and policies Based on the estimated policy gradient from the Critic, the Actor updates the policy parameter so as to obtain an optimal control policy Simulation results concerning a queuing problem illustrate that the AC learning based on AIS not only has good and stable learning performance but also has quick convergence speed

• 出版日期2010-10
• 单位中国矿业大学（北京）