A Focused Crawler for Borderlands Situation Information with Geographical Properties of Place Names

作者：Hou, Dongyang; Wu, Hao^*; Chen, Jun; Li, Ran

来源：Sustainability, 2014, 6(10): 6529-6552.

DOI：10.3390/su6106529

• focused crawler
• place name
• web information collection
• borderlands situation
• relevance calculation
• spatial relations

Place name is an important ingredient of borderlands situation information and plays a significant role in collecting them from the Internet with focused crawlers. However, current focused crawlers treat place name in the same way as any other common keyword, which has no geographical properties. This may reduce the effectiveness of focused crawlers. To solve the problem, this paper firstly discusses the importance of place name in focused crawlers in terms of location and spatial relation, and, then, proposes the two-tuple-based topic representation method to express place name and common keyword, respectively. Afterwards, spatial relations between place names are introduced to calculate the relevance of given topics and webpages, which can make the calculation process more accurately. On the basis of the above, a focused crawler prototype for borderlands situation information collection is designed and implemented. The crawling speed and F-Score are adopted to evaluate its efficiency and effectiveness. Experimental results indicate that the efficiency of our proposed focused crawler is consistent with the polite access interval and it could meet the daily demand of borderlands situation information collection. Additionally, the F-Score value of our proposed focused crawler increases by around 7%, which means that our proposed focused crawler is more effective than the traditional best-first focused crawler.

• 出版日期2014-10
• 单位国家基础地理信息中心; 中国矿业大学（北京）