Concept Discovery from Text

Transcription

1 Conept Dsovery from Text Dekang Ln and Patrk Pantel Department of Computng Sene Unversty of Alberta Edmonton, Alberta, Canada, T6G 2E8 Abstrat Broad-overage lexal resoures suh as WordNet are extremely useful. However, they often nlude many rare senses whle mssng doman-spef senses. We present a lusterng algorthm alled CBC (Clusterng By Commttee) that automatally dsovers onepts from text. It ntally dsovers a set of tght lusters alled ommttees that are well sattered n the smlarty spae. The entrod of the members of a ommttee s used as the feature vetor of the luster. We proeed by assgnng elements to ther most smlar luster. Evaluatng luster qualty has always been a dffult task. We present a new evaluaton methodology that s based on the edtng dstane between output lusters and lasses extrated from WordNet (the answer key). Our experments show that CBC outperforms several well-known lusterng algorthms n luster qualty. 1 Introduton Broad-overage lexal resoures suh as WordNet are extremely useful n applatons suh as Word Sense Dsambguaton (Leaok, Chodorow, Mller 1998) and Queston- Answerng (Pasa and Harabagu 2001). However, they often nlude many rare senses whle mssng doman-spef senses. For example, n WordNet, the words dog, omputer and ompany all have a sense that s a hyponym of person. Suh rare senses make t dffult for a oreferene resoluton system to use WordNet to enfore the onstrant that personal pronouns (e.g. he or she) must refer to a person. On the other hand, WordNet msses the user-nterfaeobet sense of the word dalog (as often used n software manuals). One way to deal wth these problems s to use a lusterng algorthm to automatally ndue semant lasses (Ln and Pantel 2001). Many lusterng algorthms represent a luster by the entrod of all of ts members (e.g., K- means) (MQueen 1967) or by a representatve element (e.g., K-medods) (Kaufmann and Rousseeuw 1987). When averagng over all elements n a luster, the entrod of a luster may be unduly nfluened by elements that only margnally belong to the luster or by elements that also belong to other lusters. For example, when lusterng words, we an use the ontexts of the words as features and group together the words that tend to appear n smlar ontexts. For nstane, U.S. state names an be lustered ths way beause they tend to appear n the followng ontexts: (Lst A) appellate ourt ampagn n aptal governor of drver's lense llegal n outlaws sth. prmary n 's sales tax senator for If we reate a entrod of all the state names, the entrod wll also ontan features suh as: (Lst B) 's arport arhbshop of 's busness dstrt fly to 's mayor mayor of 's subway outskrts of beause some of the state names (lke New York and Washngton) are also names of tes. Usng a sngle representatve from a luster may be problemat too beause eah ndvdual element has ts own dosynrases that may not be shared by other members of the luster. In ths paper, we propose a lusterng algorthm, CBC (Clusterng By Commttee), n whh the entrod of a luster s onstruted by averagng the feature vetors of a subset of the luster members. The subset s vewed as a ommttee that determnes whh other elements belong to the luster. By arefully hoosng ommttee members, the features of the entrod tend to be the more typal features of the target

2 lass. For example, our system hose the followng ommttee members to ompute the entrod of the state luster: Illnos, Mhgan, Mnnesota, Iowa, Wsonsn, Indana, Nebraska and Vermont. As a result, the entrod ontans only features lke those n Lst A. Evaluatng lusterng results s a very dffult task. We ntrodue a new evaluaton methodology that s based on the edtng dstane between output lusters and lasses extrated from WordNet (the answer key). 2 Prevous Work Clusterng algorthms are generally ategorzed as herarhal and parttonal. In herarhal agglomeratve algorthms, lusters are onstruted by teratvely mergng the most smlar lusters. These algorthms dffer n how they ompute luster smlarty. In sngle-lnk lusterng, the smlarty between two lusters s the smlarty between ther most smlar members whle omplete-lnk lusterng uses the smlarty between ther least smlar members. Average-lnk lusterng omputes ths smlarty as the average smlarty between all pars of elements aross lusters. The omplexty of these algorthms s O(n 2 logn), where n s the number of elements to be lustered (Jan, Murty, Flynn 1999). Chameleon s a herarhal algorthm that employs dynam modelng to mprove lusterng qualty (Karyps, Han, Kumar 1999). When mergng two lusters, one mght onsder the sum of the smlartes between pars of elements aross the lusters (e.g. average-lnk lusterng). A drawbak of ths approah s that the exstene of a sngle par of very smlar elements mght unduly ause the merger of two lusters. An alternatve onsders the number of pars of elements whose smlarty exeeds a ertan threshold (Guha, Rastog, Kyuseok 1998). However, ths may ause undesrable mergers when there are a large number of pars whose smlartes barely exeed the threshold. Chameleon lusterng ombnes the two approahes. K-means lusterng s often used on large data sets sne ts omplexty s lnear n n, the number of elements to be lustered. K-means s a famly of parttonal lusterng algorthms that teratvely assgns eah element to one of K lusters aordng to the entrod losest to t and reomputes the entrod of eah luster as the average of the luster s elements. K-means has omplexty O(K T n) and s effent for many lusterng tasks. Beause the ntal entrods are randomly seleted, the resultng lusters vary n qualty. Some sets of ntal entrods lead to poor onvergene rates or poor luster qualty. Bsetng K-means (Stenbah, Karyps, Kumar 2000), a varaton of K-means, begns wth a set ontanng one large luster onsstng of every element and teratvely pks the largest luster n the set, splts t nto two lusters and replaes t by the splt lusters. Splttng a luster onssts of applyng the bas K-means algorthm α tmes wth K=2 and keepng the splt that has the hghest average elemententrod smlarty. Hybrd lusterng algorthms ombne herarhal and parttonal algorthms n an attempt to have the hgh qualty of herarhal algorthms wth the effeny of parttonal algorthms. Bukshot (Cuttng, Karger, Pedersen, Tukey 1992) addresses the problem of randomly seletng ntal entrods n K-means by ombnng t wth average-lnk lusterng. Bukshot frst apples average-lnk to a random sample of n elements to generate K lusters. It then uses the entrods of the lusters as the ntal K entrods of K-means lusterng. The sample sze ounterbalanes the quadrat runnng tme of average-lnk to make Bukshot effent: O(K T n + nlogn). The parameters K and T are usually onsdered to be small numbers. 3 Word Smlarty Followng (Ln 1998), we represent eah word by a feature vetor. Eah feature orresponds to a ontext n whh the word ours. For example, threaten wth s a ontext. If the word handgun ourred n ths ontext, the ontext s a feature of handgun. The value of the feature s the pontwse mutual nformaton (Mannng and Shütze 1999 p.178) between the feature and the word. Let be a ontext and F (w) be the frequeny ount of a word w ourrng n ontext. The pontwse mutual nformaton between and w s defned as: m w, = F N ( w) F N ( w) F ( ) N

3 where N = ( ) F s the total frequeny ounts of all words and ther ontexts. A wellknown problem wth mutual nformaton s that t s based towards nfrequent words/features. We therefore multpled m w, wth a dsountng fator: F F ( w) ( w) + mn F 1 mn F ( w ), F ( ) ( w ), F ( ) + 1 We ompute the smlarty between two words w and w usng the osne oeffent (Salton and MGll 1983) of ther mutual nformaton vetors: ( ) sm w,w = 4 CBC Algorthm m m w 2 w m w m 2 w CBC onssts of three phases. In Phase I, we ompute eah element s top-k smlar elements. In our experments, we used k = 20. In Phase II, we onstrut a olleton of tght lusters, where the elements of eah luster form a ommttee. The algorthm tres to form as many ommttees as possble on the ondton that eah newly formed ommttee s not very smlar to any exstng ommttee. If the ondton s volated, the ommttee s smply dsarded. In the fnal phase of the algorthm, eah element s assgned to ts most smlar luster Phase I: Fnd top-smlar elements Computng the omplete smlarty matrx between pars of elements s obvously quadrat. However, one an dramatally redue the runnng tme by takng advantage of the fat that the feature vetor s sparse. By ndexng the features, one an retreve the set of elements that have a gven feature. To ompute the top smlar words of a word w, we frst sort w s features aordng to ther mutual nformaton wth w. We only ompute parwse smlartes between w and the words that share a hgh mutual nformaton feature wth w Phase II: Fnd ommttees The seond phase of the lusterng algorthm reursvely fnds tght lusters sattered n the smlarty spae. In eah reursve step, the Input: Step 1: Step 2: Step 3: A lst of elements E to be lustered, a smlarty database S from Phase I, thresholds θ 1 and θ 2. For eah element e E Cluster the top smlar elements of e from S usng average-lnk lusterng. For eah luster dsovered ompute the followng sore: avgsm(), where s the number of elements n and avgsm() s the average parwse smlarty between elements n. Store the hghest-sorng luster n a lst L. Sort the lusters n L n desendng order of ther sores. Let C be a lst of ommttees, ntally empty. For eah luster L n sorted order Compute the entrod of by averagng the frequeny vetors of ts elements and omputng the mutual nformaton vetor of the entrod n the same way as we dd for ndvdual elements. If s smlarty to the entrod of eah ommttee prevously added to C s below a threshold θ 1, add to C. Step 4: If C s empty, we are done and return C. Step 5: For eah element e E If e s smlarty to every ommttee n C s below threshold θ 2, add e to a lst of resdues R. Step 6: If R s empty, we are done and return C. Otherwse, return the unon of C and the output of a reursve all to Phase II usng the same nput exept replang E wth R. Output: A lst of ommttees. Fgure 1. Phase II of CBC. algorthm fnds a set of tght lusters, alled ommttees, and dentfes resdue elements that are not overed by any ommttee. We say a ommttee overs an element f the element s smlarty to the entrod of the ommttee exeeds some hgh smlarty threshold. The algorthm then reursvely attempts to fnd more ommttees among the resdue elements. The output of the algorthm s the unon of all ommttees found n eah reursve step. The detals of Phase II are presented n Fgure 1. In Step 1, the sore reflets a preferene for bgger and tghter lusters. Step 2 gves preferene to hgher qualty lusters n Step 3, where a luster s only kept f ts smlarty to all prevously kept lusters s below a fxed threshold. In our experments, we set θ 1 = 0.35.

4 Step 4 termnates the reurson f no ommttee s found n the prevous step. The resdue elements are dentfed n Step 5 and f no resdues are found, the algorthm termnates; otherwse, we reursvely apply the algorthm to the resdue elements. Eah ommttee that s dsovered n ths phase defnes one of the fnal output lusters of the algorthm Phase III: Assgn elements to lusters In Phase III, every element s assgned to the luster ontanng the ommttee to whh t s most smlar. Ths phase resembles K-means n that every element s assgned to ts losest entrod. Unlke K-means, the number of lusters s not fxed and the entrods do not hange (.e. when an element s added to a luster, t s not added to the ommttee of the luster). 5 Evaluaton Methodology Many luster evaluaton shemes have been proposed. They generally fall under two ategores: omparng luster outputs wth manually generated answer keys (hereon referred to as lasses); or embeddng the lusters n an applaton and usng ts evaluaton measure. An example of the frst approah onsders the average entropy of the lusters, whh measures the purty of the lusters (Stenbah, Karyps, and Kumar 2000). However, maxmum purty s trvally aheved when eah element forms ts own luster. An example of the seond approah evaluates the lusters by usng them to smooth probablty dstrbutons (Lee and Perera 1999). Lke the entropy sheme, we assume that there s an answer key that defnes how the elements are supposed to be lustered. Let C be a set of lusters and A be the answer key. We defne the edtng dstane, dst(c, A), as the number of operatons requred to make C onsstent wth A. We say that C s onsstent wth A f there s a one to one mappng between lusters n C and the lasses n A suh that for eah luster n C, all elements of belong to the same lass n A. We allow two edtng operatons: merge two lusters; and move an element from one luster to another. a A) b d B) e e a C) D) b d E) e a d Fgure 2. An example of applyng the transformaton rules to three lusters. A) The lasses n the answer key; B) the lusters to be transformed; C) the sets used to reonstrut the lasses (Rule 1); D) the sets after three merge operatons (Step 2); E) the sets after one move operaton (Step 3). Let B be the baselne lusterng where eah element s ts own luster. We defne the qualty of a set of lusters C as follows: dst 1 dst ( C, A) ( B, A) Suppose the goal s to onstrut a lusterng onsstent wth the answer key. Ths measure an be nterpreted as the perentage of operatons saved by startng from C versus startng from the baselne. We am to onstrut a lusterng onsstent wth A as opposed to a lusterng dental to A beause some senses n A may not exst n the orpus used to generate C. In our experments, we extrat answer lasses from WordNet. The word dog belongs to both the Person and Anmal lasses. However, n the newspaper orpus, the Person sense of dog s at best extremely rare. There s no reason to expet a lusterng algorthm to dsover ths sense of dog. The baselne dstane dst(b, A) s exatly the number of elements to be lustered. We made the assumpton that eah element belongs to exatly one luster. The transformaton proedure s as follows: 1. Suppose there are m lasses n the answer key. We start wth a lst of m empty sets, eah of whh s labeled wth a lass n the answer key. 2. For eah luster, merge t wth the set whose lass has the largest number of elements n the luster (a te s broken arbtrarly). 3. If an element s n a set whose lass s not the same as one of the element s lasses, move the element to a set where t belongs. dst(c, A) s the number of operatons performed usng the above transformaton rules on C. b e a b d e

5 Fgure 2 shows an example. In D) the luster ontanng e ould have been merged wth ether set (we arbtrarly hose the seond). The total number of operatons s 4. 6 Expermental Results We generated lusters from a news orpus usng CBC and ompared them wth lasses extrated from WordNet (Mller 1990) Test Data To extrat lasses from WordNet, we frst estmate the probablty of a random word belongng to a subherarhy (a synset and ts hyponyms). We use the frequeny ounts of synsets n the SemCor orpus (Landes, Leaok, Teng 1998) to estmate the probablty of a subherarhy. Sne SemCor s a farly small orpus, the frequeny ounts of the synsets n the lower part of the WordNet herarhy are very sparse. We smooth the probabltes by assumng that all sblngs are equally lkely gven the parent. A lass s then defned as the maxmal subherarhy wth probablty less than a threshold (we used e -2 ). We used Mnpar 1 (Ln 1994), a broadoverage Englsh parser, to parse about 1GB (144M words) of newspaper text from the TREC olleton (1988 AP Newswre, LA Tmes, and 1991 San Jose Merury) at a speed of about 500 words/seond on a PIII-750 wth 512MB memory. We olleted the frequeny ounts of the grammatal relatonshps (ontexts) output by Mnpar and used them to ompute the pontwse mutual nformaton values from Seton 3. The test set s onstruted by ntersetng the words n WordNet wth the nouns n the orpus whose total mutual nformaton wth all of ts ontexts exeeds a threshold m. Sne WordNet has a low overage of proper names, we removed all aptalzed nouns. We onstruted two test sets: S onsstng of words (m = 250) and S 3566 onsstng of 3566 words (m = 3500). We then removed from the answer lasses the words that dd not our n the test sets. Table 1 summarzes the test sets. The szes of the WordNet lasses vary a lot. For S there are 99 lasses that ontan three words or less and the largest lass ontans 3246 words. For S 3566, 78 lasses have three or less words and the largest lass ontans 1181 words. 1 Avalable at Table 1. A desrpton of the test sets n our experments. DATA SET SET TOTAL WORDS 6.2. Cluster Evaluaton We lustered the test sets usng CBC and the lusterng algorthms of Seton 2 and appled the evaluaton methodology from the prevous seton. Table 2 shows the results. The olumns are our edtng dstane based evaluaton measure. Test set S 3566 has a hgher sore for all algorthms beause t has a hgher number of average features per word than S For the K-means and Bukshot algorthms, we set the number of lusters to 250 and the maxmum number of teratons to 8. We used a sample sze of 2000 for Bukshot. For the Bsetng K-means algorthm, we appled the bas K-means algorthm twe (α = 2 n Seton 2) wth a maxmum of 8 teratons per splt. Our mplementaton of Chameleon was unable to omplete lusterng S n reasonable tme due to ts tme omplexty. Table 2 shows that K-means, Bukshot and Average-lnk have very smlar performane. CBC outperforms all other algorthms on both data sets Manual Inspeton M m Avg. Average Features # per of Features Word TOTAL CLASSES S S Table 2. Cluster qualty (%) of several lusterng algorthms on the test sets. ALGORITHM S S 3566 CBC K-means (K=250) Bukshot Bsetng K-means Chameleon n/a Average-lnk Complete-lnk Sngle-lnk Let be a luster and wn() be the WordNet lass that has the largest nterseton wth. The preson of s defned as:

6 Table 3. Fve of the 943 lusters dsovered by CBC from S along wth ther features wth top-15 hghest mutual nformaton and the WordNet lasses that have the largest nterseton wth eah luster. RANK MEMBERS TOP-15 FEATURES wn() 1 handgun, revolver, shotgun, pstol, rfle, mahne gun, sawed-off shotgun, submahne gun, gun, automat pstol, automat rfle, frearm, arbne, ammunton, magnum, artrdge, automat, stopwath 236 whtefly, pest, aphd, frut fly, termte, mosquto, okroah, flea, beetle, kller bee, maggot, predator, mte, houseplant, rket 471 supervson, dsplne, oversght, ontrol, governane, deson makng, ursdton 706 blend, mx, mxture, ombnaton, uxtaposton, ombne, amalgam, sprnkle, synthess, hybrd, melange 941 employee, lent, patent, applant, tenant, ndvdual, partpant, renter, volunteer, repent, aller, nternee, enrollee, gver blast, barrel of, brandsh, fre, pont, pull out, dsharge, fre, go off, arm wth, fre wth, kll wth, open fre wth, shoot wth, threaten wth ontrol, nfestaton, larvae, populaton, nfestaton of, spee of, swarm of, attrat, breed, eat, eradate, feed on, get rd of, repel, ward off breakdown n, lak of, loss of, assume, exerse, exert, mantan, retan, seze, tghten, brng under, operate under, plae under, put under, reman under dp n, marnate n, pour n, str n, use n, add to, pour, str, urous, elet, ethn, odd, potent, unque, unusual beneft for, are for, housng for, beneft to, serve to, fled by, pad by, use by, provde for, requre for --, gve to, offer to, provde to, dsgruntled, ndgent artfat / artfat anmal / anmate beng / beast / brute / reature / fauna at / human aton / human atvty group / groupng worker preson () wn = () CBC dsovered 943 lusters. We sorted them aordng to ther preson. Table 3 shows fve of the lusters evenly dstrbuted aordng to ther preson rankng along wth ther Top-15 features wth hghest mutual-nformaton. The words n the lusters are lsted n desendng order of ther smlarty to the luster entrod. For eah luster, we also nlude wn(). The underlned words are n wn(). The frst luster s learly a luster of frearms and the seond one s of pests. In WordNet, the word pest s urously only under the person herarhy. The words stopwath and houseplant do not belong to the lusters but they have low smlarty to ther luster entrod. The thrd luster represents some knd of ontrol. In WordNet, the legal power sense of ursdton s not a hyponym of soal ontrol as are supervson, oversght and governane. The fourth luster s about mxtures. The words blend and mx as the event of mxng are present n WordNet but not as the result of mxng. The last luster s about onsumers. Here s the onsumer lass n WordNet 1.5: addt, alohol, bg spender, buyer, lent, onert-goer, onsumer, ustomer, utter, dner, drnker, drug addt, drug user, drunk, eater, feeder, fung, head, heron addt, home buyer, unke, unky, lush, nonsmoker, patron, polyholder, purhaser, reader, regular, shopper, smoker, spender, subsrber, suker, taker, user, vegetaran, wearer In our luster, only the word lent belongs to WordNet s onsumer lass. The luster s ranked very low beause WordNet faled to onsder words lke patent, tenant and renter as onsumers. Table 3 shows that even the lowest rankng CBC lusters are farly oherent. The features assoated wth eah luster an be used to lassfy prevously unseen words nto one or more exstng lusters. Table 4 shows the lusters ontanng the word ell that are dsovered by varous lusterng algorthms from S The underlned words represent the words that belong to the ell lass n WordNet. The CBC luster orresponds almost exatly to WordNet s ell lass. K-means and Bukshot produed farly oherent lusters. The luster onstruted by Bsetng K-means s obvously of nferor qualty. Ths s onsstent wth the fat that Bsetng K-means has a muh lower sore on S ompared to CBC, K- means and Bukshot.

7 Table 4. The lusters representng the ell onept for several lusterng algorthms usng S ALGORITHMS CLUSTERS THAT HAVE THE LARGEST INTERSECTION WITH THE WORDNET CELL CLASS. CBC K-means Bukshot Bsetng K-means WordNet Class whte blood ell, red blood ell, bran ell, ell, blood ell, aner ell, nerve ell, embryo, neuron adaver, meteorte, sereton, reeptor, serum, handwrtng, aner ell, thyrod, body part, hemoglobn, red blood ell, nerve ell, urne, gene, hromosome, embryo, plasma, heart valve, salva, ovary, whte blood ell, ntestne, lymph node, sperm, heart, olon, ell, blood, bowel, bran ell, entral nervous system, spnal ord, blood ell, ornea, bladder, prostate, semen, bran, spleen, organ, nervous system, panreas, tssue, marrow, lver, lung, marrow, kdney adaver, vagna, meteorte, human body, sereton, lnng, handwrtng, aner ell, womb, ven, bloodstream, body part, eyesght, polyp, oronary artery, thyrod, membrane, red blood ell, plasma, gene, gland, embryo, salva, nerve ell, hromosome, skn, whte blood ell, ovary, sperm, uterus, blood, ntestne, heart, spnal ord, ell, bowel, olon, blood vessel, lymph node, bran ell, entral nervous system, blood ell, semen, ornea, prostate, organ, bran, bladder, spleen, nervous system, tssue, panreas, marrow, lver, lung, bone marrow, kdney pket lne, pole aademy, sphere of nfluene, bloodstream, trane, sandbox, downtown, mountan, amera, boutque, kthen snk, kln, embassy, ellblok, votng booth, drawer, ell, skylght, bookase, upboard, ballpark, roof, stadum, lubhouse, tub, bathtub, lassroom, tolet, kthen, bathroom, blood ell, bran ell, aner ell, ell, one, egg, nerve ell, neuron, red blood ell, rod, sperm, whte blood ell 7 Conluson We presented a lusterng algorthm, CBC, for automatally dsoverng onepts from text. It an handle a large number of elements, a large number of output lusters, and a large sparse feature spae. It dsovers lusters usng wellsattered tght lusters alled ommttees. In our experments, we showed that CBC outperforms several well known herarhal, parttonal, and hybrd lusterng algorthms n luster qualty. For example, n one experment, CBC outperforms K-means by 4.25%. By omparng the CBC lusters wth WordNet lasses, we not only fnd errors n CBC, but also oversghts n WordNet. Evaluatng luster qualty has always been a dffult task. We presented a new evaluaton methodology that s based on the edtng dstane between output lusters and lasses extrated from WordNet (the answer key). Aknowledgements The authors wsh to thank the revewers for ther helpful omments. Ths researh was partly supported by Natural Senes and Engneerng Researh Counl of Canada grant OGP and sholarshp PGSB Referenes Cuttng, D. R.; Karger, D.; Pedersen, J.; and Tukey, J. W Satter/Gather: A luster-based approah to browsng large doument olletons. In Proeedngs of SIGIR-92. pp Copenhagen, Denmark. Guha, S.; Rastog, R.; and Kyuseok, S ROCK: A robust lusterng algorthm for ategoral attrbutes. In Proeedngs of ICDE 99. pp Sydney, Australa. Jan, A. K.; Murty, M. N.; and Flynn, P. J Data Clusterng: A Revew. ACM Computng Surveys 31(3): Kaufmann, L. and Rousseeuw, P. J Clusterng by means of medods. In Dodge, Y. (Ed.) Statstal Data Analyss based on the L1 Norm. pp Elsever/North Holland, Amsterdam. Karyps, G.; Han, E.-H.; and Kumar, V Chameleon: A herarhal lusterng algorthm usng dynam modelng. IEEE Computer: Speal Issue on Data Analyss and Mnng 32(8): Landes, S.; Leaok, C.; and Teng, R. I Buldng Semant Conordanes. In WordNet: An Eletron Lexal Database, edted by C. Fellbaum. pp MIT Press. Leaok, C.; Chodorow, M.; and Mller; G. A Usng orpus statsts and WordNet relatons for sense dentfaton. Computatonal Lngusts, 24(1): Lee, L. and Perera, F Dstrbutonal smlarty models: Clusterng vs. nearest neghbors. In Proeedngs of ACL-99. pp College Park, MD. Ln, D Prnpar - an Effent, Broad-Coverage, Prnple-Based Parser. In Proeedngs of COLING-94. pp Kyoto, Japan. Ln, D Automat retreval and lusterng of smlar words. In Proeedngs of COLING/ACL-98. pp Montreal, Canada. Ln, D. and Pantel, P Induton of semant lasses from natural language text. In Proeedngs of SIGKDD-01. pp San Franso, CA. Mannng, C. D. and Shütze, H Foundatons of Statstal Natural Language Proessng. MIT Press. MQueen, J Some methods for lassfaton and analyss of multvarate observatons. In Proeedngs of 5 th Berkeley Symposum on Mathemats, Statsts and Probablty, 1: Mller, G WordNet: An Onlne Lexal Database. Internatonal Journal of Lexography, Pasa, M. and Harabagu, S The nformatve role of WordNet n Open-Doman Queston Answerng. In Proeedngs of NAACL-01 Workshop on WordNet and Other Lexal Resoures. pp Pttsburgh, PA. Salton, G. and MGll, M. J Introduton to Modern Informaton Retreval. MGraw Hll. Stenbah, M.; Karyps, G.; and Kumar, V A omparson of doument lusterng tehnques. Tehnal Report # Department of Computer Sene and Engneerng, Unversty of Mnnesota.s