Analysis of CDN Website Logs Based on Hadoop

作者：Song Qing^*; Wen Yujun; Gong Junpeng

来源：International Conference on Automation, Mechanical Control and Computational Engineering (AMCCE), 2015-04-24 to 2015-04-26.

• log analysis
• mass data
• Hadoop
• MapReduce

This paper designs a framework of CDN website log system based on Hadoop and a set of algorithm on the basis of user action mode excavation to analyze and process logs from searching engines. The monitor and regulation of colonies can be realized in platform monitoring modules. Under the guideline of data excavation process, this paper adopts Hadoop, an analysis tool for mass data as the experiment platform. The MapReduce reflection/excavation programming model is used. Simple and applicable HIVE from SQL and Hbase mass data pool are used to process mass logs. The writer conducts a detailed analysis on user searching action from such perspectives as topics, hits, URL order and conversational analysis to optimize platform performance and compare the system before and after the optimization. Experiment data is shown in this paper to explain that the log platform here is quite stable and efficient.

• 出版日期2015
• 单位中国传媒大学