MapReduce delay scheduling with deadline constraint

作者：Li, Hongliang; Wei, Xiaohui^*; Fu, Qingwu; Luo, Yuan

来源：Concurrency and Computation: Practice and Experience (CCPE) , 2014, 26(3): 766-778.

DOI：10.1002/cpe.3050

• MapReduce
• data-intensive
• data locality
• delay scheduling
• deadline

MapReduce programming paradigm has been widely applied to solve large-scale data-intensive problems. Intensive studies of MapReduce scheduling have been carried out to improve MapReduce system performance. Delay scheduling is a common way to achieve high data locality and system performance. However, inappropriate delays can lead to low system throughput and potentially break the original job priority constraints. This paper proposes a deadline-enabled delay (DLD) scheduling algorithm that optimizes job delay decisions according to real-time resource availability and resource competition, while still meets job deadline constraints. Experimental results illustrate that the resource availability estimation method of DLD is accurate (92%). Compared with other approaches, DLD reduces job turnaround time by 22% in average while keeping a high locality rate (88%).

• 出版日期2014-3-10
• 单位吉林大学