Seminar Data & Web Mining. Christian Groß Timo Philipp

Transcription

1 Seminar Data & Web Mining Christian Groß Timo Philipp

2 Agenda Application types Introduction to Applications Evaluation Applications 2

3 Overview Crime enforcement Crime Link Explorer Money laundering Peer Group Analysis Open Source Software Development Adolescent cigarette smoking Applications 3

4 Crime Link Explorer (1) Software developed by University of Arizona Crime investigators should be enabled to automatically conduct effective and efficient link analysis Link Analysis Identification, analysis and visualization of associations between entities : persons locations criminal incidents Applications 4

5 Crime Link Explorer (2) Three techniques included: Concept space approach Shortest path algorithm Heuristic approach Applications 5

6 Data source for Link Analysis Data source: crime incident report ( Anzeige ) Uniform Crime Report (UCR) established 1930 surveillance logs telephone records financial transactions Link: If two entities appear in the same document / log / telephone record Applications 6

7 Problems Costs much time and human effort Information overload Information buried in large volume of raw data High branching factors The number of direct links an entity has Determining the importance of links Relies heavily on domain knowledge Applications 7

8 System design GUI for visualizing founded association paths Dijkstra shortest path algorithm used for finding strong association between entities Associations are identified and extracted from the dataset using the concept space approach Heuristics, capturing domain knowledge, are used for identifying criminal associations Applications 8

9 Co-occurrence weights Applications 9

10 Concept Space Network consisting of domain specific concepts (nodes) Weighted co-occurrence relationship (links) Example: COPLINK Concepts: (nodes) Persons Organizations Locations Vehicles Link: if two concepts appear in the same criminal incidents Applications 10

11 Co-occurrence weights (1) Incident report A Location A Incident report B Person A Person A Person C Person B Person B Location B Applications 11

12 Co-occurrence weigths (2) Co-occurrence weight computed based on frequency that two persons appear together in same incident report Person A weight Person B Con: Weights computed are only a minor assistance in term of uncovering investigative leads Applications 12

13 Heuristic approach Applications 13

14 Heuristic approach three criteria: Relationship between crime type and person roles Shared addresses Shared telephone numbers Repeated co-occurrence in incident report % scale indicating the strength of associations Applications 14

15 Heuristic approach Crime type / person role (1) Construction of a matrix for each crime type Homicide Robbery Auto Theft Sexual Assault. Each matrix containing strength estimation for each role combination victim <-> witness witness <-> suspect suspect <-> victim Applications 15

16 Heuristic approach Crime type / person role (2) Table for crime type homicide: Homicide Victim Witness Suspect Arrestee Other Victim Witness... Suspect Arrestee... Other Estimation of strength of associations occurring for role combination and crime type out of every 100 incidents Heuristic score could be improved by including statistical analysis Applications 16

17 Heuristic approach Shared address / phone Important indicator for associations But: phone number often erroneous only 5 % to final weight Address more accurate than phone number 10 % to final weight Applications 17

18 Heuristic approach Co-occurrence Same idea as concept space approach But: estimation of co-occurrence weights based on empirically derived probability distribution Co-occurrence count Association probability (%) Applications 18

19 Heuristic approach Final heuristic weight P1 = crime_type / person_score P2 = shared phone score P3 = shared address score P4 = association probability based on co-occurrence counts w final Max 0.85 P P P P3, Applications 19

20 Association Path Person A Person B w 1 Person C w 2 w 4 w 3 Person D Person E w 5 Person F Applications 20

21 Association path search Logarithmic transformation done on weights w i Modified Dijkstra Algorithm used for finding strongest association path between two or more persons Applications 21

22 System Evaluation Data set: incident reports persons involved Age, gender, race, address, phone number 10 crime analysts Heuristic approach more accurate than concept space approach Heuristic approach uses domain knowledge Reduced time and effort needed for link analysis Applications 22

23 Overview Crime enforcement Link Exploration Money laundering Peer Group Analysis Open Source Software Development Adolescent cigarette smoking Applications 23

24 FAIS The Financial Crimes Enforcement Network AI System maintained by FinCEN (U.S.) Aim: detection of money laundering Applications 24

25 Information collection Information injection into DB (U.S. Customs Services Data Centre) Money Transaction over into/out of US Fill Currency Transaction Report (CTR) DB 7/19/2007 Applications 25

26 FAIS Load and prepare Data DB (U.S. Customs Services Data Centre) transaction Load Program Consolidated data Suspicious Rating Prog. Data with rating FAIS DB (Sybase) Analysis rules (336) -Subject, Accounts (linked with transactions) Link Analysis NEXPERT: GUI for investigating how result is received Allow what if statements Heuristic knowledge for text fields Rules result in individual pos./neg. evidences Bayesian transform to single rate Applications 26

27 FAIS Data Analysis Data Driven Mode User Directed Mode Apply filters Create SQL query Applications 27

28 FAIS Data Analysis (cont.) Applications 28

29 FAIS Use and Payoff Introduction 1993 Beginning 1995: 20 mio transactions 3000 subjects detected 2,5 mio accounts Beginning 1997: (see Strategy Plan of FinCEN ) 39 mio (Bank Secrecy Acts including CTR) Revealing new 3500 subjects 5,000 bank accounts of Colombian/Mexican money launderers detected Received Feedback 50% known hits 50% hits with similar behaviour 90% of leads are correct Applications 29

30 Consequences Revised Form Applications 30

31 Overview Crime enforcement Link Exploration Money laundering Peer Group Analysis Open Source Software Development Adolescent cigarette smoking Applications 31

32 OSS development phenomenon OSS := Open Source Software Hypothesis: Open Source Software development could be modeled as a self-organizing, collaborative network Collaborative network Variation of social network Edge between nodes if they are part of a collaboration Linchpins connect disparate groups into larger cluster Motivation: Better understanding of how the OSS community works IT planners are able to better calculate the risk of OSS usage Applications 32

33 OSS development (1) Recent studies showed: OSS development produces better, more bug-free software Most developers work for enjoyment and pride of being part of an successful OSS project. Not working for monetary return Collaborate from around the world Developers rarely meet face-to-face Developers are self-organized Applications 33

34 OSS development (2) OSS movement is a example of a decentralized selforganizing process. No central control or planning Threatens traditional proprietary software business Open Questions: Intellectual property rights Role of the government concerning OSS Software licensing Applications 34

35 Power Law Networks Collaborative networks often show power law distribution Examples for power law distributions: City size distribution Word ranking in languages and writing Internet Example: Applications 35

36 Data Collection and Analysis Web Crawler collected data from SourceForge (Mailing Lists, project sites, forums) from Jan 2001 to March 2002 Project number Developer id SourceForge Number of projects: (2002), (2007) Number of developers: (2002) Number of registered users: (2007) Applications 36

37 Modeling approach Modeling the OSS Community as collaborative social network Hypothesis: The OSS Movement displays power law relationships in its structure Cluster size Degree of nodes Applications 37

38 Graph modeling Node = developer Edge = work on the same project Node = projects Edge = same developer works on both projects Dev[53] emule GIMP Dev[14] Dev[75] Azureus Applications 38

39 Results Both figures show, that the two modeled networks satisfy the power law property Applications 39

40 Clustering Analysis (1) Linchpins Linchpins Applications 40

41 Number of cluster Clustering Analysis (2) Cluster size Applications 41

42 Conclusions (1) OSS developer network fits to the power law relationship OSS developer network is not a random network The graph displays preferential attachment of new nodes Initial success of a OSS project more developer join the project Important role of linchpins Attractors for other developers Facilitate the diffusion of ideas and technology between clusters Applications 43

43 Conclusions (2) Long term study needed because of high fluctuation rate of nodes Further research should be done on the OSS network Additional graph theoretic properties could be computed (cluster coefficients, network diameter, etc) Deeper understanding of how nodes join and leave Role of SourceForge? Applications 44

44 Overview Crime enforcement Link Exploration Money laundering Peer Group Analysis Open Source Software Development Adolescent cigarette smoking Applications 45

45 Adolescent Cigarette Smoking Social network theory and analysis applied to examine whether adolescents differ in prevalence of current smoking. Research project on 1092 ninth graders of 5 schools: Each choose 3 best friends (ordered by better friends first) Aim to classified each adolescent in Clique member Clique liaison Isolate Additional information provided 7/19/2007 Applications 46

46 Building Link Graph Liaisons Clique members: -Belong to group of min 3-50+% of their links within their group -Connected by some path lying entirely within the clique Clique liaisons: -2+ links with clique members/other liaisons -Not in a clique Isolates Isolates: Few/no links to other Weight of arcs = 1 if non reciprocated friendship otherwise Applications 47

47 Test data Applications 48

48 Cigarette Smoking Defined by self-report (current smoker and 1+ packs of cigarette) and carbon monoxide content in alveolar breath samples Applications 49

49 Result Smokers tent more often to be white than black (2 schools significant) come from families with mothers having lower education Applications 50

50 Additional Analysis significance in interaction at 4 schools School E significant in interactions between social position and variables grander and mother education?!? Including nonsurveyed subjects leads to 5 schools with significant relationship between social position and current smoking (not shown) Underestimation of relationship Applications 51

51 Additional Analysis (cont.) Possibility remains that isolates are integrated into peer groups outside the school social network Applications 52

52 Fiend Smoking Behaviour Isolates have more smoking friend than clique members/liaisons (1,5 4 times as many); Isolates have fewer friends than other subjects. Add attribute friend smoking to graph (ø of 3 friends - smoking/non smoking ) -> Not significant ->friend smoking is strongly related to subject smoking. Friend smoking is not a proxy for peer group social position Applications 53

53 Isolates tend to be smokers Explanation: 1. Social Isolation cause smoking 2. Smoking cause social isolation 3. No relationship between smoking & isolation (both caused by same factors) 4. Isolates are members of cliques from outside the school environment Regardless of explanation smoking is not a peer group phenomenon! Applications 54

54 Similarities / Differences between Applications Applications 55

55 Evaluation Link analysis offers a great potential to crime investigation Reduce time and human effort Domain knowledge could improve link analysis More accurate results with domain knowledge based link analysis Peer Group Analysis is a helpful tool for social network analysis Applications 56

56 7/19/2007 Applications 57