Automatically Setting Parameter-Exchanging Interval for Deep Learning

作者：Wang, Siyuan; Liao, Xiaofei^*; Fan, Xuepeng; Jin, Hai; Yao, Qiongjie; Zhang, Yu

来源：Mobile Networks and Applications, 2017, 22(2): 186-194.

DOI：10.1007/s11036-016-0740-6

• Deep learning
• Parameter server
• Parallel stochastic gradient descent
• Parameter-exchanging interval
• Resource adaptation

Parameter-server frameworks play an important role in scaling-up distributed deep learning algorithms. However, the constant growth of neural network size has led to a serious bottleneck on exchanging parameters across machines. Recent efforts rely on manually setting a parameter-exchanging interval to reduce communication overhead, regardless of the parameter-server's resource availability as well. It may face poor performance or inaccurate results for inappropriate interval. Meanwhile, request burst may occur, exacerbating the bottleneck.
In this paper, we propose an approach to automatically set the optimal exchanging interval, aiming to remove the parameter-exchanging bottleneck and to evenly utilize resources without losing training accuracy. The key idea is to increase the interval on different training nodes on the basis of the knowledge of available resources and choose different intervals for each slave node to avoid request bursts. We adopted this method to optimize the parallel Stochastic Gradient Descent algorithm, through which we successfully sped up parameter-exchanging process by eight times.

• 出版日期2017-4
• 单位华中科技大学