A multi-channel architecture for metadata management in cloud storage systems by binding CPU-cores to disks

作者：Duan, Hancong^*; Xiang, Xiaoke; Zhang, Yulu; Min, Geyong; Zhan, Wenhan; Lv, Pengcheng

来源：Concurrency and Computation: Practice and Experience (CCPE) , 2016, 28(16): 4215-4229.

DOI：10.1002/cpe.3882

• metadata
• distributed storage
• massive small files
• multi-core processor
• scalability
• channel

Metadata operations have become dominant file operations in the storage systems. In the scenarios of read-more and write-less of massive small files, the current distributed file systems suffer from the unsatisfying performance and scalability of metadata service because of random disk I/O during metadata operations. In this paper, a highly efficient metadata management architecture for cloud storage systems is proposed. The cluster design significantly improves the scalability of the system. A concept of the disk I/O channel is introduced, which is an independent data storage pipe by binding an independent CPU-core to each physical disk. In addition, a multi-channel fast key-value storage engine is proposed to provide the extremely efficient performance for the underlying storage service, which takes full advantages of multi-core processors and parallel disks I/O. Besides, a new dynamic load-balancing strategy is proposed to reduce load thrashing and improve the precision of rebalancing among the clusters. Performance measurements under a variety of benchmarks show that the metadata management is capable of handling the massive small files storage and the performance is improved significantly compared to the existing solutions.

• 出版日期2016-11
• 单位电子科技大学