A New Conjugate Gradient Method with Smoothing L-1/2 Regularization Based on a Modified Secant Equation for Training Neural Networks

作者：Li, Wenyu; Liu, Yan; Yang, Jie; Wu, Wei^*

来源：Neural Processing Letters, 2018, 48(2): 955-978.

DOI：10.1007/s11063-017-9737-9

• Feedforward neural networks
• Conjugate gradient method
• Modified secant equation
• Regularization
• Global convergence

Proposed in this paper is a new conjugate gradient method with smoothing L-1/2 regularization based on a modified secant equation for training neural networks, where a descent search direction is generated by selecting an adaptive learning rate based on the strong Wolfe conditions. Two adaptive parameters are introduced such that the new training method possesses both quasi-Newton property and sufficient descent property. As shown in the numerical experiments for five benchmark classification problems from UCI repository, compared with the other conjugate gradient training algorithms, the new training algorithm has roughly the same or even better learning capacity, but significantly better generalization capacity and network sparsity. Under mild assumptions, a global convergence result of the proposed training method is also proved.

• 出版日期2018-10
• 单位北华大学; 大连工业大学; 大连理工大学