A Network Load Sensitive Block Placement Strategy of HDFS

作者：Meng Lingjun; Zhao Wentao^*; Zhao Haohao; Ding Yang

来源：KSII Transactions on Internet and Information Systems, 2015, 9(9): 3539-3558.

DOI：10.3837/tiis.2015.09.014

• HDFS
• block placement
• network load
• imbalance
• load balance

This paper investigates and analyzes the default block placement strategy of HDFS. HDFS is a typical representative distributed file system to stream vast amount of data effectively at high bandwidth to user applications. However, the default HDFS block placement policy assumes that all nodes in the cluster are homogeneous, and places blocks with a simple Round Robin strategy without considering any nodes' resource characteristics, which decreases self-adaptability of the system. The primary contribution of this paper is the proposition of a network load sensitive block placement strategy. We have implemented our algorithm and justify it through extensive simulations and comparison with similar existing studies. The results indicate that our work not only performs much better in the data distribution but also improves write performance more significantly than the others.

• 出版日期2015-9-30
• 单位河南理工大学