A real environment oriented parallel duplicates removal approach for large scale Chinese WebPages

作者：Guo Hongzhi^*; Chen Qingcai; Xin Cong; Wang Xiaolong; Bi Ye

来源：Journal of Computational Information Systems, 2011, 7(5): 1420-1427.

• Algorithm efficiency
• Data sets
• Distributed parallel computing
• Duplicates elimination
• Effective solution
• Linux clusters
• Real environments
• Speedup
• Web page

Aiming at the problems that existing Chinese webpage duplicate removal approaches are restricted by the page scale and the algorithm efficiency, we propose an efficient distributed parallel duplicates elimination approach based on Linux cluster. This paper not only solves the problems of memory limitations and large computation caused by the huge data scale, but also researches into the time-series problems in the distributed parallel computing and gives an effective solution. Experimental results on 10 million webpage dataset show that the proposed approach can deal with duplicates from massive web pages well and truly.

• 出版日期2011