Wei Zheng College of Science, Hebei North University, Zhangjiakou , Hebei, China

Transcription

1 do: / An Improved CHI Feature Selecton Method for Chnese Text Classfcaton We Zheng College of Scence, Hebe North Unversty, Zhangjakou , Hebe, Chna Hadong Wang Lbrary of Hebe Insttute of Archtecture and Cvl Engneerng, Zhangjakou , Hebe, Chna Dongmn Wu* Dean's Offce, Chongqng Normal Unversty, ChongQng, , Chna *Correspondng Abstract We Proposed a knd of feature selecton method named ICHI based on mproved CHI. Through the classfed experment,the result showsthat feature extracton effect of CHI method s better than the tradtonal CHI s when them s used to select features n SVM and KNN classfcaton, and the ICHI method can enhance theaccuracy n text classfcaton and t s fttedto extract feather. Key words:feature Selecton,CHI, Text Classfcaton 1. INTRODUCTION The text classfcaton s to assgn automatcally a new document nto one or more predefned classes based on ts contents. Wth the development of WWW, n recent years, text categorzaton (TC) has become one of the key technques for handng and organzng text data, especally for Web page and document. Therefore, t s very necessary and meanngful to study the key technology of text categorzaton for mprovng the speed and accuracy of categorzaton. A major dffculty of text categorzaton s the hgh dmensonalty of the orgnal feature space.feature selecton s an mport method to reduce the amount of feature n text categorzaton, and ts goals are mprovng classfcaton effectveness and computatonal effcency. Currently, the feature selecton method s prncple of operaton s that t wll compute and score for each feature word usng statstcal knowledge, accordng to sort the feature words, then t select some feature whose score s hgher to act fnal document feature. Some wellknown methods are document frequency(df), nformaton gan (IG),expected cross entropy (ECE),the weght of evdence of text (WET), χ2statstc (CHI) and so on (Feng and Zheng,2011; Yang and Pederson, 1997), and t s hghly desrable to reduce the feature space wthout the loss of classfcaton accuracy. In recent years, a growng number of statstcal classfcaton methods and data mnng learnng technques have been appled n ths feld. Feature selecton methods were analyzed n lterature (Shenand Lu,2006), the performance of feature extracton were compared though expermental, the expermental results show that the IG and CHI, have a better effect.ig has a large amount of calculaton relatve to other several methods, and CHI method has become a common feature selecton n text categorzaton model for ts low tme complexty, easy to understand, convenent applcaton (Kou and Ca,2007; Zhengand Wang,2007).Therefore, t has a realstc sgnfcance that we study n-depth the advantages and dsadvantages of CHI to fnd an effectve mprovement of the model to mprove ts applcaton effcency. In ths paper we propose a feature selecton method based on mproved CHI,named ICHI derved from the CHI orgnal defnton. The ICHI overcome the shortcomngs of CHI, such as CHI method exst no ndependence hypothess between words and categores and the hgh computatonal complexty drawback. Experments on Chnese text data collecton collected by the Fudan Unversty show the performance of ICHI method. The rest of ths paper s organzed as follows. Secton 2 descrbes the feature selecton method CHI commonly used. Secton 3 studes the shortcomng of CHI and Secton 4 gves an mproved CHI method named ICHI.Secton5 dscusses the classfer usng n experment to compare ICHI wth other text feature selecton methods, and presents the experment s results and analyss. In the last secton, we gve the concluson and future work. 117

2 2.CHI CHI-SQUARE STATISTIC (CHI) CHI s used to measure the nterdependences between classes and feature t, The ch statstc method measures the lack of ndependence between the term and the category. If term t and category are ndependent, then the CHI s 0. c [ P( t, c ) P( t, c ) P( t, c ) P( t, c )] CHI ( t, c ) (1) P( t) p( t ) P( c ) P( c ) If there are nclasses,then each term value wll have n correlaton value, the average value calculaton for a category as follows: 2 ( ) n CHI t P( c)log CHI ( t, c ) (2) 1 3.SHORTCOMINGS OF CHI Ch-square statstc (CHI) concept from the of Contngency table test, t can be used to measure the statstcal correlaton between the characterstcs of t and category c, as shown n table 1. Table T-C(Term-Category)Contngency table t c j A c j t C D When we measure a related degree between the feature t and category c usng statstcal methods, we should suppose a hypothess between t and c n accordance wth ch-square dstrbuton wth frst-order degrees of freedom. A s the document numbers that the documents contans t and belongs to the category c.b s s the document numbers that the document s contans t and don t belongs to the category c. C s the document numbers that the documents don t contans t and belongs to the category c.d s the document numbers that the document s don t contans t and don t belongs to the category c.n s total numbers of the expermental data. The correlaton degree between termt and category c calculated as shown n the formula (3): B 2 2 N( AD BC) ( A C)( B D)( A B)( C D ) (3) CHI methods mantan the followng dsadvantages on the formula (2) and (3): (1) There are two condtons n the correlaton of categores and feature,whch one s postve, and the another s negatve.as formula (3) descrpton :f AD-BC>0, attrbute and category are relevant, whch feature document may belong to a certan category, and CHI statstc s greater when a document contans the characterstc more lkely to belong to a category. Conversely, f AD-BC<0, t s negatvely between attrbute and category that the feature appear n a related document, but the document may not belong to a category whch appears frequently n other categores, but the CHI statstc values may be bg, whch s a shortcomng of the CHI. (2) the experment show that CHI defect s lke to look for the low -frequency word. In the actual data set, there are a number of low-frequency words and only a mnorty of these low-frequency words has a strong correlaton wth the category. Most of the low-frequency words are nose words, whch should not be choose nto feature subset. A low frequency nose word only appearng n few categores, are negatvely related to most categores, but tradtonal CHI statstcal methods often gve a hgh ratng, whch affect the classfcaton results. When the data set don t uneven dstrbuton, the dropng n t s Classfcaton accuracy s obvous. Because of the shortcomng of CHI method, we propose two knds of mprovement measures to construct a modfed feature selecton method based on CHI. 4.A FEATURE SELECTION METHOD ICHI BASED ON IMPROVED CHI Though the analyss, we wll mproved the tradtonal CHI statstc methods, and remove the negatve stuaton between category and feature.tthe mproved algorthm s as follows: 118

3 P( t, c ) (, ) (, ) (, ) 0 P t c P t c P t c AD BC CHI ( t, c ) (4) 0 AD BC 0 The CHI values n mprovng the CHI statstc methods, s the total sum on all categores of statstc.in other words, when the CHI value s calculated, we only need consderng the postve stuatons. For negatve correlaton stuatons between the category and feature words,we reset the CHI statstc to 0 and don t consder t n the summaton statstcs, n addton. From related research(luo,2004; Yang and Lu,1999), ths artcle summarzes, when we meet the followng 3 demands,t s helpful to select features.these requrements are three demands that they are Dsperson Informaton,Concentraton Informaton and Frequency Informaton factors. (1) concentraton Informaton: when a feature only appear one or few text,not appear many categores, the feature s helpful to text classfcaton and s power to representaton category. (2) Dsperson Informaton : If a feature n a unform appears n the category text,t suggest that t has a strong assocaton wth the class, that s, a feature appear n a large amounts of texts n a category, the more dspersed, t s more helpful to category. (3)Frequency Informaton: The average number that a feature appear n documents of a category s large, and such feature s value for classfcaton. Concentraton, dsperson, frequency calculaton formula s as follows: (1) Concentraton Informaton (2)Dstrbuton Informaton number of documents that category C contan feature t CI (5) the total number of documents of Tranng set contanng feature t numbers of documents that category C contan feature t DI (6) numbers of documents of category C (3)Frequency Informaton numbers of words that category C contan feature t FI (7) numbers of documents of category C Ths paper proposed a new feature selecton method namecd ICHI (Improved CHI) based on the mproved CHI, whch concentraton, dsperson and frequency are ntroduced nto evaluaton functon,and ICHI assessment functon s as follows: ( P( t, c ) (, ) (, ) (, )) ( ) 0 P t c P t c P t c CI DI FI AD BC ICHI ( t, c ) (8) 0 AD BC 0 5. EXPERMENT AND ANALYSIS We use IG, CHIand ICHI method to select useful features,and carry out classfcaton experment n order to verfy the effect of feature selecton Data Collectons The expermental data used n ths paper s from Chnese natural language processng group n Department of Computng and Informaton technology n Fudan unversty(luo Yong,2004).The tranng corpus s tran.rar whch has 20 categores ncludes about 9804 documents and test.rar ncludes about 9833 documents s used for test. We just choose some of the documents for our experments because of consderng the effcency of the algorthm. Table 1 shows the specfc quantty of samples n each category we chose. Table 2. Expermental Data Quantty of tranng documents Quantty of testng documents C1-computer C2-Envornmet C3-Educaton C4-Medcal C5-Traffc In all

4 Classfer SVM and KNN classferwere wdely applcaton n n text classfcaton experment n the, and has made good classfcaton effect (Yang and Lu,1999; Duand Xao,2002), whch s currently more outstandng of classfer. The SVM classfcaton s acheved n experment by support vector machne t specfc usng LIBSVM package (Ma,Land Teng,2008), whch s develop and desgn by Tawan Unversty Ln Zhren Dr. KNN classfer s achevedthrough programmng. 5.3.Performance Measure To evaluate the performance of a text classfer, we use F1 measure put forward byrjsbergen(1979)(yang,1999). Ths measure combnes recall and precson as follows: Recall Precson number of correct postve predctons number of postve examples number of correct postve predctons number of postve predctons (9) (10) 2 Recall Pr ecson F1 (11) ( recall Pr ecson) 5.4.Results and Analyss We have selected feature experment and the feature dmenson s the length from the 1000 to Expermental results have shown n Fgure 1,Fgure2, and the best result for each feature selecton method as shown n table 2. IG CHI ICHI F 1 测试值 Values % % % % % % % % % numbers of features Fgure 1.The performance of three feature selecton methods on SVM F 1 测试值 Values % % % % % % % % % IG CHI ICHI numbers of features Fgure 2. The performance of three feature selecton methods on KNN 120

5 Table 3.Method of optmum classfcaton results compare classfcaton Recall Precson F 1 value SVM IG % % % CHI % % % ICHI % % % KNN IG % % % CHI % % % ICHI % % % Through Fgure 1,we can see that the F 1 values for CHI s greater than IG and CHI whose classfcaton curve s smooth, whch result s consstent to lterature (Luo Yong, 2004).When the dmenson number of feature s 4000, F 1 value of classfcaton correspondng IG curve reached %. The effect of classfcaton s obvously better than CHI and IG, specally the dmenson number of feature s 1000 to 3000, When features dmenson number 1000 to 3000, and 3000 to 7000, there were no major transtons on ICHI curve node.when the feature dmenson equal to 3000 and 7000,the curve turn pont and ts correspondng category F 1 values of more than 98.9%, especally when the feature dmenson s 3000, F 1 value for % Fgure 1 maxmum pont.the classfcaton effect correspondng to ICHI s obvously better than the other two feature selecton methods. Fgure 2 s a classfcaton effect fgure corresponds to three selecton methods n usng KNN classfcaton algorthm. When dmenson number of feature take 1000, the classfcaton effect of CHI s better than IG and ICHI methods. When dmenson number of feature take 2000 to 7000, F 1 curve of ICHI method corresponds to classfcaton s obvously above CHI and IG curve, and F 1 value take of maxmum value % on 6000 dmenson.the data n table 3 are extreme values of the recall, precson and F 1 values usng SVM and KNN classfer correspondng to 3 types of feature selecton methods. The maxmum F 1 values for ICHI s greater than IG and CHI. Fgure 1, Fgure 2 and table 2 have shown that a new feature selecton method ICHI based on mproved CHI could extract some category features that s vald n the Chnese text categorzaton.usng ICHI method can mprove the accuracy of classfcaton, and has a good job stablty n the feature extracton. 6.CONCLUSIONS Ths paper has proposed an mproved feature selecton method based on CHI, named ICHI. ICHI mplemented three prncples whch are Concentraton Informaton, Dstrbuton Informaton and Frequency nformaton. The experment has shows that ICHI s an effectve method to extract category characterstcs for feature selecton, and t can effectvely mprove the performance of text categorzaton. In the future, we wll ontnue work on the representaton model of 3 prncples of and effectve combnaton of nfluence factors. Acknowledgements The work was supported the Natonal Natural Scence Foundaton of Chna ( ), and Natural Scence Foundaton of Hebe Provnce (F ). REFERENCES Feng Guohe, Zheng We(2011) Revew of Chnese Automatc Word Segmentaton,Lbrary and nformaton servce, 5(2),pp Ymng Yang,Pederson JO, A(2006) Comparatve Study on Feature Selecton n Text Categorzaton,Proc. Conf. on Machne learnng,pp Shen Hong,Lu Baolang(2006) Comparson and Improvments of Feature Extracton Methods for TextCategorzaton, Computer Smulaton, 6(3),pp KouSu-lng,CaQngsheng(2007) Research on FeatureSelectonnChnese Text Classfcaton,Computer smulaton,12(9),pp Zheng We,Wang Ru(2007) Comparatve Study of Feature Selecton n Chnese Text Categorzaton,Journal of Hebe North Unversty,23(6),pp Xong Zhongyang., Zhang Pengzhao,Zhang Yufang(2008) Improved approach to CHI n feature extracton,computer Applcatons,24(2),pp Ta Dey,WangJun(2010) Improved Feature Weghtng Algorthm for Text Categorzaton,Computer engneerng,26(17),pp

6 Luo Yong(2004) The research on mproved mutual nformaton feature selecton method for text categorzaton,fujan computer,28(4),pp Ymng Yang, Xn Lu(1999) A Re-Examnaton of Text Categorzaton Methods,Proceedngs of ACMSIGIR Conf.on Research and Development n Informaton Retreval, pp Ma Janbn,L Yng, Teng Gufa(1999) The comparson studes on the algorthm of KNN and SVM for chnese text Classfcaton,Journal of agrcultural Unversty of Hebe, 2008(3),pp Du Yunq,Xao Shbn(1999) Study on Chnese Text Categorzaton Based on Support Vector Machne,Computer Engneerng,28(11),pp Ymng Yang(1999) An evaluaton of statstcal approaches to text categorzaton,journal of Informaton Retreval,25(1),pp