A high-performance distributed file system for large-scale concurrent HD video streams

作者：Duan, Hancong^*; Zhan, Wenhan; Min, Geyong; Guo, Hui; Luo, Shengmei

来源：Concurrency and Computation: Practice and Experience (CCPE) , 2015, 27(13): 3510-3522.

DOI：10.1002/cpe.3528

• streaming media
• high concurrency
• high throughput
• continuous storage
• energy saving

With the rapid development of intelligent transportation technology, high-definition data storage, and processing of massive amounts of video surveillance have become key issues. When thousands of high-bit-rate video streaming concurrently writes, disk I/O throughput becomes a bottleneck. In addition, this leads to serious energy consumption and disk abrasion. To solve these problems, a new distributed file system for high concurrent and high-bit-rate writing is designed. It combines an optimized data storage model, efficient metadata management, and exquisite disk schedule mechanism. The optimized data storage model uses a file pre-allocation strategy and multiple-stream input modulating technology to convert the randomly concurrent writes on the disk into sequential writes; the metadata management provides an efficient means of retrieving the specified data; and the dual-partition schedule mechanism can ensure the disk stability with less abrasion. Through this distributed file system, the disk I/O throughput can be saturated in a high concurrent writing environment. The performance evaluation results demonstrate that the I/O throughput of a normal 7200RPM SATA III disk in our scheme can be stabilized at 150MB/s, easily to support 300 concurrent high-definition video streams (4Mbit/s each). The distributed file system with eight commodity servers can afford the ability of supporting 8000 high-definition video streams concurrently writing, which is far greater than the existing video surveillance storage solutions.

• 出版日期2015-9-10
• 单位电子科技大学