A pragmatic model for new Chinese word extraction

作者：Zhang Haijun^*; Huang Heyan; Zhu Chaoyong; Shi Shumin

来源：6th International Conference on Natural Language Processing and Knowledge Engineering, NLP-KE 2010, 2010-08-21 to 2010-08-23.

DOI：10.1109/NLPKE.2010.5587846

• Computational efficiency
• Formal description
• Left (right) entropy
• New words extraction
• Repeat

This paper proposed a pragmatic model for repeat-based Chinese New Word Extraction (NWE). It contains two innovations. The first is a formal description for the process of NWE, which gives instructions on feature selection in theory. On the basis of this, the Conditional Random Fields model (CRF) is selected as statistical framework to solve the formal description. The second is an improved algorithm for left (right) entropy to improve the efficiency of NWE. By comparing with baseline algorithm, the improved algorithm can enhance the computational speed of entropy remarkably. On the whole, experiments show that the model this paper proposed is very effective, and the F score is 49.72 in open test and 69.83 in word extraction respectively, which is an evident improvement over previous similar works.

• 出版日期2010
• 单位北京理工大学