Recap. CS276A Text Retrieval and Mining. The Curse of Dimensionality. Today s Topics: Clustering 2. Hierarchical Agglomerative Clustering (HAC)

Transcription

1 CS276A Text Retreval and Mnng Leture 14 Reap Why luster douments? For mprovng reall n searh applatons For speedng up vetor spae retreval Navgaton Presentaton of searh results k-means bas teraton At the start of the teraton, we have k entrods. Eah do assgned to the nearest entrod. All dos assgned to the same entrod are averaged to ompute a new entrod; thus have k new entrods. The Curse of Dmensonalty Why doument lusterng s dffult Whle lusterng looks ntutve n 2 dmensons, many of our applatons nvolve 10,000 or more dmensons Hgh-dmensonal spaes look dfferent: the probablty of random ponts beng lose drops qukly as the dmensonalty grows. One way to look at t: n large-dmenson spaes, random vetors are almost all almost perpendular. Why? Next lass we wll menton methods of dmensonalty reduton mportant for text Today s Tops: Clusterng 2 Herarhal lusterng Agglomeratve lusterng tehnques Evaluaton Term vs. doument spae lusterng Mult-lngual dos Feature seleton Labelng Herarhal Clusterng Buld a tree-based herarhal taxonomy (dendrogram) from a set of unlabeled examples. anmal vertebrate nvertebrate fsh reptle amphb. mammal worm nset rustaean One opton to produe a herarhal lusterng s reursve applaton of a parttonal lusterng algorthm to produe a herarhal lusterng. Herarhal Agglomeratve Clusterng (HAC) Assumes a smlarty funton for determnng the smlarty of two nstanes. Starts wth all nstanes n a separate luster and then repeatedly ons the two lusters that are most smlar untl there s only one luster. The hstory of mergng forms a bnary tree or herarhy. 1

2 A Dendogram: Herarhal Clusterng HAC Algorthm Dendrogram: Deomposes data obets nto a several levels of nested parttonng (tree of lusters). Clusterng of the data obets s obtaned by uttng the dendrogram at the desred level, then eah onneted omponent forms a luster. Start wth all nstanes n ther own luster. Untl there s only one luster: Among the urrent lusters, determne the two lusters, and, that are most smlar. Replae and wth a sngle luster Herarhal Clusterng algorthms Agglomeratve (bottom-up): Start wth eah doument beng a sngle luster. Eventually all douments belong to the same luster. Dendrogram: Doument Example As lusters agglomerate, dos lkely to fall nto a herarhy of tops or onepts. Dvsve (top-down): Start wth all douments belong to the same luster. Eventually eah node forms a luster on ts own. Does not requre the number of lusters k n advane Needs a termnaton/readout ondton The fnal mode n both Agglomeratve and Dvsve s of no use. d1 d2 d3 d4 d5 d1,d 2 d4,d 5 d3,d4,d 5 d3 Closest par of lusters Many varants to defnng losest par of lusters Center of gravty Clusters whose entrods (enters of gravty) are the most osne-smlar Average-lnk Average osne between pars of elements Sngle-lnk Smlarty of the most osne-smlar (sngle-lnk) Complete-lnk Smlarty of the furthest ponts, the least osnesmlar Herarhal Clusterng Key problem: as you buld lusters, how do you represent the loaton of eah luster, to tell whh par of lusters s losest? Euldean ase: eah luster has a entrod = average of ts ponts. Measure nterluster dstanes by dstanes of entrods. 2

3 Sngle Lnk Agglomeratve Clusterng Sngle Lnk Example Use maxmum smlarty of pars: sm(, ) = max sm( x, y) x, y Can result n straggly (long and thn) lusters due to hanng effet. Approprate n some domans, suh as lusterng slands: Hawa lusters After mergng and, the smlarty of the resultng luster to another luster, k, s: sm (( ), k ) = max( sm(, k ), sm(, k )) Complete Lnk Agglomeratve Clusterng Complete Lnk Example Use mnmum smlarty of pars: sm(, ) = mn sm( x, y) x, y Makes tghter, spheral lusters that are typally preferable. After mergng and, the smlarty of the resultng luster to another luster, k, s: sm (( ), k ) = mn( sm(, k ), sm(, k )) Computatonal Complexty In the frst teraton, all HAC methods need to ompute smlarty of all pars of n ndvdual nstanes whh s O(n 2 ). In eah of the subsequent n 2 mergng teratons, t must ompute the dstane between the most reently reated luster and all other exstng lusters. Sne we an ust store unhanged smlartes In order to mantan an overall O(n 2 ) performane, omputng smlarty to eah other luster must be done n onstant tme. Else O(n 2 log n) or O(n 3 ) f done navely Key noton: luster representatve We want a noton of a representatve pont n a luster Representatve should be some sort of typal or entral pont n the luster, e.g., pont ndung smallest rad to dos n luster smallest squared dstanes, et. pont that s the average of all dos n the luster Centrod or enter of gravty 3

4 Example: n=6, k=3, losest par of entrods Outlers n entrod omputaton d6 d4 Can gnore outlers when omputng entrod. What s an outler? Lots of statstal defntons, e.g. moment of pont to entrod > M some luster moment. d5 d1 d3 d2 Centrod after seond step. Centrod Say 10. Outler Centrod after frst step. Group Average Agglomeratve Clusterng Computng Group Average Smlarty Use average smlarty aross all pars wthn the merged luster to measure the smlarty of two lusters. 1 sm(, ) = ( Compromse between sngle and omplete lnk. Two optons: 1) Averaged aross all ordered pars n the merged luster Averaged over all pars between the two orgnal lusters Some prevous work has used one of these optons; some the other. No lear dfferene n effay r r r r x ( ) y ( ) : y x r r sm( x, y) Assume osne smlarty and normalzed vetors wth unt length. Always mantan sum of vetors n eah luster. r r s( ) = x r x Compute smlarty of lusters n onstant tme: r r r r ( s( ) + s( )) ( s( ) + s( )) ( + sm(, ) = ( + )( + 1) ) Effeny: Medod As Cluster Representatve The entrod does not have to be a doument. Medod: A luster representatve that s one of the douments For example: the doument losest to the entrod One reason ths s useful Consder the representatve of a large luster (>1000 douments) The entrod of ths luster wll be a dense vetor The medod of ths luster wll be a sparse vetor Compare: mean/entrod vs. medan/medod Exerse Consder agglomeratve lusterng on n ponts on a lne. Explan how you ould avod n 3 dstane omputatons - how many wll your sheme use? 4

5 Effeny: Usng approxmatons In standard algorthm, must fnd losest par of entrods at eah step Approxmaton: nstead, fnd nearly losest par use some data struture that makes ths approxmaton easer to mantan smplst example: mantan losest par based on dstanes n proeton on a random lne Random lne Term vs. doument spae So far, we lustered dos based on ther smlartes n term spae For some applatons, e.g., top analyss for ndung navgaton strutures, an dualze : use dos as axes represent (some) terms as vetors proxmty based on o-ourrene of terms n dos now lusterng terms, not dos Term vs. doument spae Cosne omputaton Constant for dos n term spae Grows lnearly wth orpus sze for terms n do spae Cluster labelng lusters have lean desrptons n terms of noun phrase o-ourrene Easer labelng? Applaton of term lusters Sometmes we want term lusters (example?) If we need do lusters, left wth problem of bndng dos to these lusters Mult-lngual dos E.g., Canadan government dos. Every do n Englsh and equvalent Frenh. Must luster by onepts rather than language Smplest: pad dos n one language wth dtonary equvalents n the other thus eah do has a representaton n both languages Axes are terms n both languages Feature seleton Whh terms to use as axes for vetor spae? Large body of (ongong) researh IDF s a form of feature seleton Can exaggerate nose e.g., ms-spellngs Better s to use hghest weght md-frequeny words the most dsrmnatng terms Pseudo-lngust heursts, e.g., drop stop-words stemmng/lemmatzaton use only nouns/noun phrases Good lusterng should fgure out some of these Maor ssue - labelng After lusterng algorthm fnds lusters - how an they be useful to the end user? Need pthy label for eah luster In searh results, say Anmal or Car n the aguar example. In top trees (Yahoo), need navgatonal ues. Often done by hand, a posteror. 5

6 How to Label Clusters Labelng Show ttles of typal douments Ttles are easy to san Authors reate them for quk sannng! But you an only show a few ttles whh may not fully represent luster Show words/phrases promnent n luster More lkely to fully represent luster Use dstngushng words/phrases Dfferental labelng But harder to san Common heursts - lst 5-10 most frequent terms n the entrod vetor. Drop stop-words; stem. Dfferental labelng by frequent terms Wthn a olleton Computers, lusters all have the word omputer as frequent term. Dsrmnant analyss of entrods. Perhaps better: dstntve noun phrase Evaluaton of lusterng Perhaps the most substantve ssue n data mnng n general: how do you measure goodness? Most measures fous on omputatonal effeny Tme and spae For applaton of lusterng to searh: Measure retreval effetveness Approahes to evaluatng Anedotal User nspeton Ground truth omparson Cluster retreval Purely quanttatve measures Probablty of generatng lusters found Average dstane between luster members Mroeonom / utlty Anedotal evaluaton Probably the ommonest (and surely the easest) I wrote ths lusterng algorthm and look what t found! No benhmarks, no omparson possble Any lusterng algorthm wll pk up the easy stuff lke partton by languages Generally, unlear sentf value. User nspeton Indue a set of lusters or a navgaton tree Have subet matter experts evaluate the results and sore them some degree of subetvty Often ombned wth searh results lusterng Not lear how reproduble aross tests. Expensve / tme-onsumng 6

7 Ground truth omparson Take a unon of dos from a taxonomy & luster Yahoo!, ODP, newspaper setons Compare lusterng results to baselne e.g., 80% of the lusters found map leanly to taxonomy nodes How would we measure ths? Subetve But s t the rght answer? There an be several equally rght answers For the dos gven, the stat pror taxonomy may be nomplete/wrong n plaes the lusterng algorthm may have gotten rght thngs not n the stat taxonomy Ground truth omparson Dvergent goals Stat taxonomy desgned to be the rght navgaton struture somewhat ndependent of orpus at hand Clusters found have to do wth vagares of orpus Also, dos put n a taxonomy node may not be the most representatve ones for that top f Yahoo! Mroeonom vewpont Anythng - nludng lusterng - s only as good as the eonom utlty t provdes For lusterng: net eonom gan produed by an approah (vs. another approah) Strve for a onrete optmzaton problem Examples reommendaton systems lok tme for nteratve searh expensve Evaluaton example: Cluster retreval Ad-ho retreval Cluster dos n returned set Identfy best luster & only retreve dos from t How do varous lusterng methods affet the qualty of what s retreved? Conrete measure of qualty: Preson as measured by user udgements for these queres Done wth TREC queres Evaluaton Sm-Ranked vs. Cluster-Ranked Compare two IR algorthms 1. send query, present ranked results 2. send query, luster results, present lusters Experment was smulated (no users) Results were lustered nto 5 lusters Clusters were ranked aordng to perentage relevant douments Douments wthn lusters were ranked aordng to smlarty to query 7

8 Relevane Densty of Clusters Bukshot Algorthm Another way to an effent mplementaton: Cluster a sample, then assgn the entre set Bukshot ombnes HAC and K-Means lusterng. Frst randomly take a sample of nstanes of sze n Run group-average HAC on ths sample, whh takes only O(n) tme. Use the results of HAC as ntal seeds for K- means. Overall algorthm s O(n) and avods problems of bad seed seleton. Uses HAC to bootstrap K-means Cut where You have k lusters Bsetng K-means Exerses Dvsve herarhal lusterng method usng K-means For I=1 to k-1 do { Pk a leaf luster C to splt For J=1 to ITER do { } Use K-means to splt C nto two sub-lusters, C 1 and C 2 Choose the best of the above splts and make t permanent} } Stenbah et al. suggest HAC s better than k-means but Bsetng K-means s better than HAC for ther text experments Consder runnng 2-means lusterng on a orpus, eah do of whh s from one of two dfferent languages. What are the two lusters we would expet to see? Is agglomeratve lusterng lkely to produe dfferent results to the above? Is the entrod of normalzed vetors normalzed? Suppose a run of agglomeratve lusterng fnds k=7 to have the hghest value amongst all k. Have we found the hghest-value lusterng amongst all lusterngs wth k=7? Resoures Satter/Gather: A Cluster-based Approah to Browsng Large Doument Colletons (1992) Cuttng/Karger/Pedersen/Tukey Data Clusterng: A Revew (1999) Jan/Murty/Flynn A Comparson of Doument Clusterng Tehnques Mhael Stenbah, George Karyps and Vpn Kumar. TextMnng Workshop. KDD Resoures Intalzaton of teratve refnement lusterng algorthms. (1998) Fayyad, Rena, and Bradley Salng Clusterng Algorthms to Large Databases (1998) Bradley, Fayyad, and Rena 8