Automated Chat Thread Analysis: Untangling the Web

Transcription

1 Automated Chat Thread Analyss: Untanglng the Web Dr. Sowmya Ramahandran, Randy Jensen, Osar Basara, Tamtha Carpenter Stottler Henke Assoates, In. San Mateo, CA Todd Dennng AFRL/HEA Nells AFB, NV Lt Shaun Sullon AFRL Mesa, AZ ABSTRACT As networked dgtal ommunatons prolferate n mltary operatonal ommand and ontrol (C2, hat messagng s emergng as a preferred ommunatons method for team oordnaton. Chat room logs provde a potentally rh soure of data for analyss n after-aton revews, affordng onsderable nsght nto the deson-makng proesses among the tranng audene. The multtaskng nature of these types of operatons, and the large number of hat hannels and partpants lead to multple, parallel threads of dalogs that are tghtly ntertwned. It s neessary to dentfy and separate these threads to faltate analyss of hat ommunaton n support of team performane assessment. Ths presents a sgnfant hallenge as hat s prone to nformal language usage, abbrevatons, and typos. Tehnques for onventonal language analyss do not transfer very well. Few nroads have been made n taklng the problem of dalog analyss and top deteton from hat messages. In ths paper, we wll dsuss the applaton of natural language tehnques to automate hat log analyss, usng an AOC team tranng exerse as the soure of data. We have found t neessary to enhane these tehnques to take nto onsderaton the spef haratersts of hat-based C2 ommunatons. Addtonally, our doman of nterest provdes other data soures besdes hat that an be leveraged to mprove lassfaton auray. We wll desrbe how suh onsderatons have been folded nto tradtonal data analyss tehnques to address ths problem and dsuss ther performane. In partular, we explore the problem of automatally detetng ontent-based oherene between messages. We present tehnques to address ths problem and analyze ther performane n omparson wth usng dstngushng keywords provded by subjet matter experts. We dsuss the lessons learned from our results and how t mpats future work. Dstrbuton A: Approved for publ release. Unlmted dstrbuton. As part of 88ABW , 18 June ABOUT THE AUTHORS Dr. Sowmya Ramahandran s a researh sentst at Stottler Henke Assoates, a small busness dedated to provdng nnovatve Artfal Intellgene solutons to real-world problems. Dr. Ramahandran's nterests fous on ntellgent tranng and eduaton tehnology nludng ntellgent tutorng and ntellgent synthet agents for smulatons. She s also nterested n ssues of motvaton and metaognton. Experene wth mltary and prvate ndustry gves Dr. Ramahandran a unque perspetve on the needs and requrements of the ultmate end-users and ther onstrants. She ontrbutes expertse n AI, nstrutonal systems, probablst reasonng, and knowledge management. She has developed ITSs for a range of tops nludng readng omprehenson, hgh-shool Algebra, helopter plotng, and healthare domans. She has partpated n workshops organzed by the Learnng Federaton, a dvson of the Federaton of Ameran Sentsts, to lay out a roadmap for rtal future researh and fundng n the area of ITSs and vrtual patent smulatons. She has developed a general-purpose authorng framework for rapd development of ITSs, whh s urrently beng used to develop an ntellgent tutor tranng Navy Tatal Aton Offers. She has also developed tools and tehnologes for tranng emergeny frst responders. Todd Dennng s a tranng researh nvestgator for AOC tranng researh for the Ar Fore Researh Laboratory s Warfghter Readness Researh Dvson n Mesa, AZ and an nstrutor/subjet matter expert for dynam and delberate plannng tranng wth the 505th Operatons Squadron, Nells AFB, NV. He has extensve experene n 2010 Paper No Page 1 of 11

2 fghter and ar operatons plannng and operatons to nlude ombat operatons n Southwest Asa and the Paf regon. Randy Jensen s a group manager at Stottler Henke Assoates, In., workng n tranng systems sne He has developed numerous Intellgent Tutorng Systems for Stottler Henke, as well as authorng tools, smulaton ontrols, after aton revew tools, and natural language analyss methods. He s urrently leadng projets to develop an embedded tranng Intellgent Tutor for the Army, an after aton revew toolset for the Ar Fore, and an authorng tool for vrtual tranng demonstratons for the Army. He holds a B.S. wth honors n symbol systems from Stanford Unversty. Osar Basara s a software engneer at Stottler Henke Assoates, In. Hs nterests nlude tranng systems, authorng tools, and user nterfae desgn. He holds an M.Eng. n Eletral Engneerng from Cornell Unversty and an M.A. n Mathemats from the Unversty of Calforna at Berkeley. Lt Shaun Sullon s a behavoral sentst assgned to the Ar Fore Researh Laboratory s 711th Human Performane Wng, Warfghter Readness Researh Dvson n Mesa, AZ. He s the government projet manager for Stottler Henke Assoates, In., Small Busness Innovatve Researh effort to develop an automated performane assessment after-aton revew tool that analyzes hat ommunatons among teams. Lt Sullon earned hs Bahelor of Sene degree n Behavoral Senes from the Unted States Ar Fore Aademy n Colorado Sprngs, CO Paper No Page 2 of 11

3 Automated Chat Thread Analyss: Untanglng the Web Dr. Sowmya Ramahandran, Randy Jensen, Osar Basara, Tamtha Carpenter Stottler Henke Assoates, In. San Mateo, CA INTRODUCTION Ramahandran et. al desrbed the need for tools to faltate the analyss of eletron ommunatons among teams. Communaton optons lke hat and emal offer benefts over tradtonal rado and are beomng a vtal part of team nteratons. Ths provdes an unpreedented opportunty for team ommunaton analyss. Text-based ommunatons an be logged and analyzed to study the team s performane. If there were falures or undesrable events n an operaton, the logs an be examned to determne the ontrbuton of ommunaton falures to the stuaton. In a broader ontext, mnng hat and other text message-based ommunatons s gong to be of nreasng mportane n the future. Instant messagng and hat are beomng mportant tools for team ommunatons. The trend s rapdly towards nreasng adopton of hat-based ommunaton even wthn the mltary. Automated analyss of hat wll grow n value for varous purposes suh as: 1. Analyzng and mprovng busness ommunatons, 2. Detetng top trends, 3. Analyzng messages streams for ntellgene and ounter-ntellgene analyss. Our researh to date s foused on the use of hat for mltary plannng operatons. The multtaskng nature of these types of operatons, and the large number of hat hannels and partpants lead to multple, parallel threads of dalogs that are tghtly ntertwned. It s neessary to dentfy and separate these threads to faltate analyss of hat ommunaton n support of team performane assessment. Ths presents a sgnfant hallenge as hat s prone to nformal language usage, abbrevatons, and typos. Tehnques for onventonal language analyss do not transfer very well. Todd Dennng AFRL/HEA Nells AFB, NV Todd.dennng.tr@nells.af.ml Lt Shaun Sullon AFRL Mesa, AZ Shaun.sullon@mesa.afm.af.ml Despte all the advanes n the feld of Natural Language Proessng (Mannng and Shütze, 1999, understandng the semants of language s stll a bg hallenge for omputers. The objetve of our researh s to explore the extent to whh hat-based ommunatons an be analyzed to extrat useful nformaton wthout a deep semant understandng of the messages. We fous on the use of statstal and rule-based tehnques that wll analyze messages based on surfae features suh as word ourrenes and orrelatons. Ramahandran et. al. dsussed a ombnaton of doman-spef and doman-ndependent tehnques to separate hat data nto threads of related onversaton around a top. The partular doman of applaton s the Ar and Spae Operatons Center (AOC plannng operatons and as suh we were nterested n separatng out onversaton threads relatng to dfferent targets or mssons. Ths ntal approah onssted of 1. Usng Subjet Matter Expert (SME-provded keywords to assoate messages wth spef mentons, and 2. Usng a proess of temporal pattern reognton to dentfy talk-response dyads n the onversatons to dentfy oherent sets of related onversatons. Keyword-based assoaton s a rual step n the proedure that leads to hghly relable assoatons sne t s provded by experts famlar wth the dstngushng haraterst of eah msson. However, ths human-n-the-loop soluton does not aheve our researh objetve of developng a ommunaton analyss tool that requres mnmal human nput. To mnmze the human effort requred to ondut afteraton revews, t s desrable to automate hat analyss to the extent pratal. Ths paper wll dsuss tehnques for automatally dentfyng keywords that dstngush the dfferent mssons Paper No Page 3 of 11

4 BACKGROUND Ths work s n support of the researh at the Ar Fore Researh Lab at Mesa, AZ amed at mprovng team tranng outomes and developng exerse vsualzaton and debrefng tools that wll help tranees and traners. As a targeted tranng doman, the Ar Fore Researh Laboratory s Tranng Researh Exerse (T-REX provdes a ontrolled researh envronment to nvestgate team performane dynams n an ar and spae operatons enter. The envronment allows msson-ready warfghters to prate ther assgned dutes usng real-world systems n a senaro desgned to test the full spetrum of desons and oordnaton requred n operatonal plannng. The sute of systems nludes ollaboratve plannng tools, nludng hat rooms. As the warfghters ondut msson dutes, researhers ollet nformaton on a varety of performane areas, leveragng hat as the omplementary real-tme ommunaton mode n assoaton wth the sute of ollaboratve tools and shared stuaton awareness nputs avalable n an AOC. The researh objetves pursued n a T-REX exerse are to: 1 Develop mmersve senaros to stmulate full team partpaton; 2 Develop tools to apture and valdate team performane measures whle ondutng jont fore plannng for knet and non-knet effets; and, 3 Develop a synhronzed sute of after-aton revew dsplays and tools to effetvely ommunate performane bak to the team mmedately after a tranng sesson. The researh approah used n determnng how to analyze and dsplay nformaton follows the operatonal plannng methodology lad out n jont and USAF dotrne. The ntator for plannng s normally a problem statement n the form of ntellgene data or operatonal data reported to the team. The ntatng report typally establshes a segregated plannng approah to address the problem. The team then examnes the problem n sequene wth other plannng tasks or a sub-team may be tasked to examne the ssue n parallel wth other team atvtes. In many ases, plannng may be nterrupted and take on an nterleaved harater. When a tranng sesson ends, tranees need to be able to see eah problem n solaton, as well as n ontext wth other workload. The solaton approah allows the team to revew atual proess versus dotrne, whle the ontext of workload offers nsght nto tme delays, dstratons, errant nformaton soures, and overall ogntve effort. The most sgnfant hallenge to ondutng an effetve after aton revew of operatonal plannng s to solate proesses effently for onsumpton by dfferent members or subgroups wthn the tranng audene. Problems n operatonal warfare rarely nvolve an entre audene, sne the team s omposed of ndvduals wth unque and non-overlappng areas of expertse. At the leadershp level of the team, the deson makers must be able to trak and revew desons n full vew of the nformaton avalable at the tme to understand how well they ated on t. Plannng spealsts nvolved n a proess wll also want to segregate and revew nformaton pertanng only to the proess n queston. The spealsts not nvolved n a proess wll want the revew to move qukly enough to get to the next pont n tme where they are nvolved. After aton revew tools must help an nstrutor to sort and assoate nformaton wth a unque proess and be able to dsplay nformaton ogently to dentfy key areas that postvely or negatvely affeted team and ndvdual performane. Ths s true n the general sense, rrespetve of the form that exerse data takes. Where hat logs are one of the prmary soures of data ndatng performane, tools for revewng multple hat logs n tandem beome rtal. Intellgent Dagnost Assstant (IDA s ntended as a mxed ntatve soluton that leverages the strengths of the mahne and the human. The strength of the mahne les n data management, organzaton, flterng, presentaton, and automated analyss for smple keyword-based and temporal-based patterns. The strength of the human les n seletng analyss rtera and performng hgh-level, bg-pture analyss. For example, wth the TREX exerses, nstrutors have expressed a strong need for a tool that wll lassfy the hat data aordng to mssons, further assoate hat segments wth dfferent phases of a proess, and provde omplementary vsualzaton that wll larfy the ommunaton flow wthn eah proess. Rather than supplantng nstrutonal tasks, the goal s to faltate them, so that nstrutors wll be able to use ther expertse effently to dentfy the tranng ponts and supportng data they wsh to emphasze. Thus, the goal s to develop a tool that serves as a ogntve ad to nstrutors developng an After-Aton Revew (AAR. Our approah to automatng hat analyss for the purposes of developng an nstrutor s tool dvdes nto apabltes to support two prmary atvtes: 1. Assoaton and flterng: In order to nrease the speed and effeny of puttng together an AAR, automated natural language analyss and pattern reognton tehnques produe a prelmnary assoaton between hat messages and spef mssons of the exerse. Ths assoaton s the bakbone of a flterng apablty that nstrutors 2010 Paper No Page 4 of 11

5 use to narrow the sope of the hat data they wll be revewng as they explore spef lnes of nqury nto tranee desons. 2. Vsualzaton and browsng: Even wth a fltered set of hat data, t s stll a tme onsumng task to revew synhronous onversaton streams n multple hat rooms and develop an understandng of the overall flow to dentfy performane ndators. Ths s the motvaton for a talored browsng apablty that an nstrutor an use to revew proess-spef ommunatons and vsualze hronologal relatonshps rossreferened wth mssons. Typally, ommunatons regardng a partular target wll flow aross multple hat rooms, so synhronous browsng s a key feature. Addtonally, the results of assoatons and flterng an be refleted n the browsng envronment as ues durng the revew proess. For example, keywords related to a msson proess that were deteted n the flterng step wll often be of nterest to an nstrutor as hghlghted terms whle browsng. The nstrutor uses these tools to fous on proessspef ommunatons and draw ther own onlusons about how the team s ommunaton helped or hndered ahevng the msson objetves. The vsualzaton aspet of ths tool was dsussed n detal n a prevous paper on IDA (Ramahandran et. al., Ths paper fouses on IDA s approah for assoatng hat messages wth spef exerse mssons. FIRST PASS: RULE-BASED ASSOCIATIONS We wll desrbe our ntal algorthm and dsuss the refnements to t to address ts varous lmtatons. The ntal rule-based analyss, frst desrbed n Ramahandran et. al. apples the followng four rules n sequene: Rule 1: In ths doman, eah msson/target s assgned a unque dentfaton number (ID. Tranees sometmes, but not always, wll refer to ths ID whle talkng about a msson. When they do, ths makes t easy to assoate the hat messages wth a msson/target. IDA makes one pass through the data set to dentfy those messages that have explt referenes. These messages form the ore set upon whh subsequent rules buld. Rule 2: The next pass uses msson-spef keywords to lassfy hat messages. The keywords are provded by SMEs n a onfguraton fle pror to analyss. The fat that eah msson has a set of unque dentfers (e.g., msson numbers, ode names for plaes or people, target types s leveraged to tag hat messages. Typally ths pass results n a smaller but stll sgnfant number of untagged messages. Rule 3: There are some types of temporal patterns that an be deteted wth relable auray wthout the need to understand the ontent of utteranes. An example s reognzng the pattern of a turn-by-turn nteraton between two people n the same room (e.g. A says somethng to B and 3 mnutes later B says somethng to A and nferrng that they belong to the same top thread. Makng an assumpton of dalog oherene, one an say wth a hgh degree of onfdene that suh onversaton dyads refer to the same top thread. The message lassfatons dentfed usng the keywordbased approah s used as the bass to further dentfy and tag suh pars of messages. Rule 4: Fnally, loalty nfluene s used to attempt to lassfy remanng unlassfed hat lnes. For eah suh lne, IDA examnes ts neghborng messages and fnds the most ommon msson assoaton, weghted by dstane of the neghbor from the lne. If the ombned nfluene of all the messages wthn that wndow that are assoated wth ths msson s over a threshold, the hat lne s also assgned to that msson. Although ths rule has been mplemented, t has not been analyzed suffently to gauge ts usefulness. Ths wll be done as a part of future researh. Outome Results ndate that the lassfatons resultng from ths approah are moderately aurate. Table 1 shows the lassfaton auray of these rules on data from real hat logs from one of the T-REX exerse sessons. We provde three related measures of auray. Preson s a measure of the number of data tems lassfed orretly as a fraton of the total number of data that were assgned a lassfaton. Reall s a measure of the number of data tems lassfed orretly as a fraton of the total number data tems that atually belong to those lasses (as spefed by ground truth nformaton. The F-sore s harmon mean of these two measures. All of these measures range between 0 and 1, wth 1 sgnfyng the best auray. All of the results reported here use a T-REX data set wth 631 hat lnes and 20 mssons. All auraes reported n ths paper are averages of the preson, 2010 Paper No Page 5 of 11

6 reall, and F-sore measures for eah msson. The data set was hand labeled wth message-msson assoatons by an SME. Ths formed the gold standard aganst whh IDA s output was evaluated. Table 1. Auray of Classfaton Resultng From Intal Rules Data Set IDA Classfaton Auray Preson Reall F-Sore T-REX These rules allow for multple lassfatons of the same hat message (.e. eah message an be assgned to multple lasses. Ths leads to a sgnfant number of false postves (refleted n a lower preson sore. For IDA, however, false postves are preferable to false negatves. Flterng data onservatvely s better than flterng out messages that are related to the msson of nterest. The rules result n a hgh reall auray whh means there are few false negatves. However, mprovng preson wll be an ongong objetve. A more rtal lmtaton of ths approah s ts relane on hand oded keywords. The followng setons desrbe our ongong efforts to elmnate ths need. CLUSTERING Our frst approah to dentfyng related messages based on statstal analyss was a tehnque alled lusterng. Ths s an establshed and popular Artfal Intellgene tehnque for automatally groupng data wthout human nput. Our hypothess was that ths tehnque would lead to hgh-value top based lusters that an be used, n addton to the rules mentoned above, to separate the ommunatons relatng to dfferent mssons. The lusterng approah has the advantage of not requrng that the tranng data be labeled by hand. We ntrodued a lusterng step for assoatng messages wth plannng proesses based on the term frequeny nverse doument frequeny (TF-IDF smlarty measure presented n (Adams and Martell, Ths measure determnes smlarty by the number of overlappng words between two messages, weghted by the unqueness of the words. That s, under ths sheme, ommon words suh as at, the, et. wll have lower weght and therefore smaller nfluene on the smlarty measure. Unque words wll ontrbute more heavly. The frst pass of the modfed algorthm s the same as before. It looks for msson IDs to reate an ntal set of message-msson assoatons. Durng the next pass, t uses a nearest neghbor approah to buld message lusters based on the TF-IDF smlarty measure. Wthn eah luster, t looks for messages that have been lassfed to mssons based on Step 1. Clusters wth messages that have been assoated wth more than one msson are elmnated as not beng relevant to the analyss (.e. the smlarty between the messages n the luster s not pertnent to the tops of nterest. Clusters wth at least one message assoated wth a msson are dentfed and all the untagged messages n eah luster are assgned to the msson. Consder an example wth 8 hat messages, U1 through U8, and three mssons, M1 through M3. The followng table shows the proess assoatons after the frst pass. Table 2. Intal Assoatons between Message and Mssons Message U1 U2 U3 U4 U5 U6 U7 U8 Proess M1 M2 M3 Now, suppose the nearest neghbor approah dentfes message lusters as shown n the followng table. Table 3. Results of Clusterng Clusters Messages Mssons C1 U1, U2, U6, U7 U1<->M1 U6<->M3 C2 U4, U5 M2 C3 U3, U8 Cluster C1 s elmnated from any further proessng sne t has assoatons wth multple mssons. Cluster C3 s also elmnated beause t has no msson assoatons. That leaves C2. In ths luster, U5 s assoated wth M2 and therefore U4 s also tagged wth M2. The resultng proess assoatons after the seond pass wll be as shown n Table 4. After the seond pass, the remanng passes to make assoatons based on request-response dyads and 2010 Paper No Page 6 of 11

7 message proxmty are made. The rules for these subsequent passes have not hanged. Table 4. Message-Msson Assoatons after Clusterng Message U1 U2 U3 U4 U5 U6 U7 U8 Proess M1 M2 M2 M3 Addtonally, we had to make one modfaton to the tradtonal lusterng algorthm desrbed above. We observed that whle the lusters dentfed by ths method have an dentfably unfyng top, ths top sometmes s tangental to the mssons beng dsussed. For example, the algorthm may luster together messages whh are smlar n that they all talk about tmes on target (TOTs. Ths s not nterestng from the perspetve of dentfyng dfferent msson-related dsussons, as there s nothng msson-spef about TOTs. So we nluded a varaton where messages are only ompared to others wthn a spefed tme wndow. Ths mproved the relevane of the generated lusters. Outome Ths approah stll needs some messages to be assgned to a msson aordng to Rule 1 above (.e. based on referenes to the atual msson/target IDs. The lusterng omponent uses ths seed set to then lassfy other messages that are n the same lusters as these seeds. Whle testng the lusterng algorthm, we observed that there are hat onversatons about mssons where the tranees do not ever menton the target IDs. The lusterng approah fals under these ondtons. Whle we have not performed a quanttatve evaluaton of the utlty of the lusterng omponent, ndatons are that t provdes some utlty and therefore, we wll retan t n the mx. However, ths wll fal to dentfy keywords under the ondtons mentoned above. LEVERAGING OTHER RELATED INFORMATION SOURCES The doman provdes another related data soure that ould be usefully exploted. All tranees use a database system alled Jont Automated Deep Operatons Coordnaton System (JADOCS to reord rtal nformaton about the varous mssons, suh as target ntellgene, operatonal orders et. A very ommon prate s to opy over messages from hat streams to the JADOCS database (DB as annotatons. Ths results n a set of hat messages stored n the JADOCS DB wth defnte msson assoatons that an be mned to learn msson-spef dentfers. However, ths data an be sparse and t s an empral queston f t s suffent to for IDA to learn aurate lassfatons.. Our next enhanement was to use Naïve Bayes (Langley 1995 lassfers, desrbed below, that were traned on the message-msson assoatons found n the JADOCS. Ths s done n plae of Rule 2 of the orgnal approah. The remanng rules are appled as before. Naïve Bayes Classfers wth Normalzaton Our approah uses a separate Naïve Bayes lassfer for eah msson lassfaton. A normalzaton proess s then appled to the results of these lassfers to obtan the msson lassfaton probabltes for a hat message. We have a set of msson lassfatons or lasses: C = { 1, 2,, C } (1 These lasses are used to label a set of tranng hat messages or douments: D = {d 1, d 2,, d D } (2 That s, the tranng doument d j s ether assoated wth lass or not assoated wth lass. The set of all the words ontaned n the tranng douments s the voabulary: V = {w 1, w 2,, w V } (3 Consder a new test doument d to be lassfed. Bayes Rule an be appled to ompute the posteror probablty : 2010 Paper No Page 7 of 11

8 d (4 d To ompute the pror probablty, we smply dvde the number of tranng douments assoated wth lass by the total number of tranng douments: D j 1 j (5 D where j = 1 or 0 dependng f tranng doument d s assoated wth lass. Next, to ompute the lkelhood d, we use the Naïve Bayes assumpton that eah word n a doument s ndependent of the ourrene of other words to get the followng: V d [ ( (1 (1 ( ] t B 1 t P wt B t P wt (6 where B t = 1 or 0 dependng f doument d ontans word w t. To ompute w t, we an dvde the number of tranng douments ontanng w t and assoated wth lass by the total number of tranng doument assoated wth lass. But to avod probabltes of 0 or 1, the dvson s prmed: 1 2 D j 1 jt ( wt D B j 1 P (7 Fnally, the pror probablty d n Bayes Rule s a onstant that wll anel durng normalzaton. Now two normalzaton steps are performed. Frst, we normalze between the and, where s the event of not lass, to get: P norm 1 ( d d d (8 j j P P ( ( norm1 norm ( 2 d (9 C P ( k 1 norm 1 k d The probablty P ( s used to determne f norm 2 the test doument d s assoated wth lass. Table 5 shows the auray of ths approah on one data set. For omparson, we have shown the results of applyng the rule-based approah desrbed n the Seton FIRST PASS: RULE-BASED ASSOCIATIONS, wth and wthout keywords. (Note: all these versons nluded the lusterng step desrbed above. The results reported are for the same data set. Note that the reall auray suffers dramatally when no keywords are provded (omparng rows 1 and 2 of the table. The preson sore s mproved however n the absene of keywords. Ths ndates that the keywords, whle overng more of the true postves, also result n mslassfyng more negatve examples. The Bayesan lassfer (row 3 that replaes the keywords mproves the reall auray sgnfantly whle also mprovng preson. Thus t s sgnfantly better than usng no keywords (row 1 at all, both n terms of dentfyng more of the true postves and mslassfyng fewer of the negatves. Compared to hand-oded keywords (row 2 approah, the Bayesan approah results n better preson but worse reall. Thus t does not mslassfy as many negatves but fals to lassfy as many true postves. Thus the automated approah, whle not as effetve as usng hand-oded keywords, s sgnfantly better than not usng any keywords at all. Furthermore, t offers the onvenene of not requrng the traner or a subjet matter expert to supply the keywords. Analyzng IDA s lassfaton on a dfferent data set, we observed that the tranees experened onsderable onfuson between two targets and mxed up ther target data desrptons several tmes. As a result, IDA was not able to dstngush between these two mssons very aurately. Whle ths s an ssue for IDA, t may also be possble to rase a flag when suh onfusons are deteted as they may ndate useful tranng ponts. Ths possblty wll be explored n the future. Seond, we normalze between all these normalzed probabltes to get: 2010 Paper No Page 8 of 11

9 Rule- Based Table 5. Auray of JADOCS-Based Approah Classfaton Method Hand Coded Automated Keywords Keyword Deteton IDA Auray Preson Reall F-Sore Yes Yes No Yes No Yes Yes No No LESSONS LEARNED 1. Data fuson,.e. olletng and analyzng data from multple soures s a powerful way of harnessng omplementary nformaton for analyss. IDA s able to tap nto data from the JADOCS DB, whh, whle not extensve, ontans rually salent nformaton that s needed for analyzng the larger database of hat messages. 2. Tops, whle separate, are sometmes hghly orrelated. For example, exerses may have one msson dedated to a Hgh-Value-Indvdual (HVI, another to the HVI s loaton, and another to a planned operaton targetng the HVI. Sometmes these dstntons are genunely neessary; at other tmes they are just artfats of a msunderstandng on the part of tranees about ommunaton and bookkeepng protools. Whatever the reason, ths makes top dentfaton hallengng n the absene of a deeper semant nterpretaton. Statstal tehnques for natural language analyss, suh as the ones dsussed n ths paper, are lmted n ths respet. 3. For the above-mentoned reason, we have observed that even humans famlar wth the detals of the exerse fnd t dffult to assoate hat messages to the relevant top threads (.e. mssons. Ths makes evaluatng the performane of IDA a hallenge as there are no aurate ground truth lassfatons to serve as standards for omparson. We wll have to use an alternate approah to perform a robust evaluaton of IDA s analyss. 4. Fnally, we note that there s room for mprovement n IDA s lassfaton approah. The F-Sores for all versons are lower than desred. Ths s largely due to hgh numbers of false postves. Improvng ths wll be an mportant fous n the near future. The dsusson ponts above pont out the hallenges to ahevng ths goal. We wll work wth the AOC traners to determne how to best approah top onfusons suh as the one mentoned above. Tryng to dfferentate between hghly orrelated mssons s a sgnfant hallenge for automated tehnques. However, f the exstene of suh orrelatons turns out to an artfat of nsuffent understandng of the proess on the part of tranees, IDA an be engneered to dentfy suh onfusons and turn them nto tranng opportuntes. Ths wll be an mportant fous of ths researh gong forward. Fnally, we have foused our attenton to date largely on automatng keyword-based lassfaton. Gong forward, we wll also analyze the performane of rules 3 and 4, study ther ontrbuton to lassfaton auray, and tune them. RELATED WORK Prevous related researh nvolvng mult-party dalog analyss has nluded muh work to haraterze spoken nteratons n mult-party meetngs, soal strutures, and ollaboratve learnng envronments. The most relevant work s beng done by the Cogntve Agent that Learns and Organzes (CALO projet, a jont effort between SRI and Stanford Unversty s Center for the Study of Language and Informaton. (Zmmermann 2006, and (Tur 2008 desrbe efforts wthn the CALO projet to support mult-party meetngs wth transrpton, aton tem extraton, and, n some ases, software ontrol suh as doument retreval and dsplay updatng. (Nekrasz 2004 desrbe an arhteture n whh the spoken onversaton between meetng partpants s proessed usng automat speeh reognton tehnques, and grounded aganst the artfat beng produed (e.g., a shedule, a budget and the drawngs made on an eletron whteboard. All of these nputs are used to reate an eletron verson of the artfat. Although experments wth dalog models from spoken nteratons are transferable to researh wth hat ommunatons, there are also unque hallenges wth the hat medum Paper No Page 9 of 11

10 Muh hat-related researh has foused on the nherent ommunaton artfats of the medum, suh as the emergene of onventonal abbrevatons, emotons, and other ommon stylst prates. To a lesser degree, some researh has yelded methods and tools to analyze or vsualze hat ommunaton patterns. Most requre a odng step arred out by a human reader to tag messages or expltly dentfy dependenes before analyss takes plae n any automated form. (Cakr 2005 studed methods for assessng team problem solvng wth a hat envronment and shared workspae. Essentally ths employed a struture for organzng messages and dentfyng nstanes of nteratons between two, three, or more partpants as well as ndes for fators lke ntatve. Ths s useful for learnng researh observatons about how level and type of partpaton ontrbute to team dynams and ollaboraton effetveness. (Sh 2006 ntrodue a oneptual framework for thread theory, whh suggests an approah for sortng out dfferent hat threads based on top or theme, and for haraterzng defnng features suh as lfe, ntensty, magntude, and level of partpaton. (Herrng 2006 desrbes VsualDTA, a tool desgned to generate a vsualzaton of a hat onversaton that has been manually oded. In ths vsualzaton, messages are plotted n a desendng tree, wth temporal spang represented on one axs, and semant dvergene represented on the other. The tool also aommodates the possblty of ompletely new top threads appearng wthn the hat stream, resultng n new trees. Ths s useful for soal nteraton researh, where plots of ommunaton patterns reveal behavoral features. (Adam and Martell, 2008 used the TF-IDF measure dsussed earler to dentfy top threads n hat onversatons. Ther approah used only lusterng whereas we have sute of other tehnques to help the proess. Whereas they were onerned wth detetng tops n general publ hat sessons that are not foused on any partular doman, our objetves are narrower. We are onerned prmarly wth hat onversatons that are our wthn mltary team tranng exerses. Ths gves us the beneft of leveragng hat protools, doman-spef voabulary, and other data soures to help refne our tehnque. ACKNOWLEDGEMENTS The researh effort desrbed n ths paper s sponsored by the Ar Fore Researh Laboratory. REFERENCES Adams, P.H. and Martell, C. H Top Deteton and Extraton n hat. In the Proeedngs of the IEEE Conferene on Semant Computng. Santa Clara, CA. Cakr, M., Xhafa, F., Zhou, N., and Stahl, G. (2005, Thread-based analyss of patterns of ollaboratve nteraton n hat, paper presented at the Conferene of Artfal Intellgene for Eduaton (AIEd 05, Amsterdam, Netherlands. Herrng, S. C., & Kurtz, A. J. (2006. Vsualzng dynam top analyss, Proeedngs of CHI'06. New York: ACM Press. Langley, Pat (1995. Elements of Mahne Learnng.Morgan Kaufman Seres n Mahne Learnng Mannng, Chrstopher and Shüteze, Hnrh (1999. Foundatons of Statstal Natural Language Proessng. The MIT Press Nekrasz, J., Gruensten, A., & Cavedon, L. (2004. Mult-human dalog understandng for assstng artfat-produng meetngs. In Proeedngs of the 20th Internatonal Conferene on Computatonal Lngusts (COLING. Ramahandran, S., R. Jensen, O. Basara, T. Carpenter, T. Dennng, S. Sullon (2009 After Aton Revew Tools For Team Tranng wth Chat Communatons. Proeedngs of the Industry/Interserve, Tranng, Smulaton & Eduaton Conferene (I/ITSEC Sh, S., Mshra, P., Bonk, C. J., Tan, S., & Zhao, Y. (2006. Thread theory: A framework appled to ontent analyss of synhronous omputer medated ommunaton data, Internatonal Journal of Instrutonal Tehnology and Dstane Learnng, 3(3, Tur, G., Stolke, A., Voss, L., Dowdng, J., Favre, B., Fernandez, R., Frampton, M., Frandsen, M., Frederkson, C., Graarena, M., Hakkan-Tür, D., Kntzng, D., Leveque, K., Mason, S., Nekrasz, J., 2010 Paper No Page 10 of 11

11 Peters, S., Purver, M., Redhammer, K., Shrberg, E., Ten, J., Vergyr, D., & Yang, F. (2008. The CALO meetng speeh reognton and understandng system, In Proeedngs of the 2008 IEEE Workshop on Spoken Language Tehnology. Zmmermann, M.; Lu, Y.; Shrberg, E. & Stolke, A. (2006. Jont Segmentaton and Classfaton of Dalog Ats n Multparty Meetngs. In Proeedngs of IEEE ICASSP, Toulouse, Frane ( Paper No Page 11 of 11