A novel incremental parallel web crawler based on focused crawling

作者：Huang Qiuyan; Li Qingzhong^*; Yan Zhongmin; Fu Hong

来源：Journal of Computational Information Systems, 2013, 9(6): 2461-2469.

• Focused crawler
• Incremental crawler
• Multi objective decision making
• Parallel crawler
• Update frequency graph

With the tremendous growth of the Web, it has become a huge challenge for the all-purpose singleprocess crawlers to locate the resources that are precise and relevant in an appropriate amount of time, so more enhanced and convincing algorithms are in demand. In this paper, a novel incremental parallel Web crawler based on focused crawling is proposed, which can crawl the Web pages that are relevant to multiple pre-defined topics concurrently. Furthermore, to solve the issue of URL distribution, a compound decision model based on multi-objective decision making method is introduced, which considers multiple factors synthetically such as load balance, relevance and so on; and to solve the issue of update frequency of local repository decision, a update frequency graph model is presented, in which the graph is constructed dynamically according to the update frequency of Web pages. The extensive experiments show that our proposed system can acquire high quality, high relevance and high freshness Web information efficiently.

• 出版日期2013