N-grams based feature selection and text representation for Chinese Text Classification

作者：Wei Zhihua^*; Miao Duoqian; Chauchat Jean Hugues; Zhao Rui; Li Wen

来源：International Journal of Computational Intelligence Systems, 2009, 2(4): 365-374.

DOI：10.1080/18756891.2009.9727668

• Chinese text classification
• n-gram
• feature selection
• text representation weight

In this paper, text representation and feature selection strategies for Chinese text classification based on n-grams are discussed. Two steps feature selection strategy is proposed which combines the preprocess within classes with the feature selection among classes. Four different feature selection methods and three text representation weights are compared by exhaustive experiments. Both C-SVC classifier and Naive bayes classifier are adopted to assess the results. All experiments are performed on Chinese corpus TanCorpV1.0 which includes more than 14,000 texts divided in 12 classes. Our experiments concern: (1) the performance comparison among different feature selection strategies: absolute text frequency, relative text frequency, absolute n-gram frequency and relative n-gram frequency; (2) the comparison of the sparseness and feature correlation in the "text by feature" matrices produced by four feature selection methods; (3) the performance comparison among three term weights: 0/1 logical value, n-gram frequency numeric value (TF) and Tf*idf value.

• 出版日期2009-12
• 单位同济大学