An Optimal Solution of Storing and Processing Small Image Files on Hadoop

作者：Su, Qiubin; Lu, Lu^*; Feng, QiuYan

来源：9th International Conference on Brain-Inspired Cognitive Systems, BICS 2018, 2018-07-07 To 2018-07-08.

DOI：10.1007/978-3-030-00563-4_63

• Distributed computations
• Distributed storage
• Hadoop distributed file system (HDFS)
• Hadoop distributed file systems
• Map-reduce
• Self-dined storage and IO format
• Small images files
• Social network services

The rapid development of the Internet, especially mobile Internet, makes it much easier for people to make social contacts online. Nowadays people tend to spend more and more time on social network service, and produce a lot of image files. This brings a challenge to traditional standalone framework on handing the continued increasing image files. Therefore, it is advisable to find a new way to settle the challenge. Hadoop is a notable, widely-used project for distributed storage and computations with high efficiency, data integrity, reliability and fault tolerance. Hadoop Distributed File System and MapReduce are two primary subprojects respectively for big data storage and computations. However, Hadoop does not provide any interface for image processing. Moreover, both Hadoop Distributed File System and MapReduce have trouble in processing large amount of small files, which result in decreasing efficiency of files access and distributed computations. This prevents us from performing images processing actions on Hadoop. In view of this, this paper proposes a new method to optimize the storage of small image files on Hadoop and self-defines an input/output format to enable Hadoop to process image files.

• 出版日期2018
• 单位华南理工大学