depend on knowing in advance what the fraud perpetrators are atlarnpting. Sampling and Analysis File Preparation Nonnal Behavior Samples

Transcription

1 Use of Data Enterprise Miner & Multivariate Statistical Analysis to Build Fraud Detection Models Shiao-ping Lu, Lucid Consulting Group, Palo Alto, CA ABSTRACT The increasing number of checking account fraud cases and the dollar loss reported at Wells Fargo Bank evoked the need to build an early-detection program at the point r:4 transaction Since the fraud rate is relati\lely small ~ared to milions of transactions per month, it is both desirable and warranted to explore new predictive techniques and to provide eariy warnings to the fraud prevention unit This paper will address data sampling, variable construction and model construclion Several approaches taken in the modeling with the utilization of Data Enterprise Miner include logistic regression, treeanalysis and neural nelw()r1( The accuracy of prediction, misclassification costs and robustness of various models will be compared Other multivariate statistical methods such as factor analysis in variable dimension reduction will also be discussed NTRODUCTON Background n spite the presence of ~ fraud detection programs, the reported cases d checking account fraud and millions d dollar loss are rising at Wells Fargo Bank Majority of the fraud detection is operated in batch processing (after-the-fact) and the detection mechanism is more or less "chasing the fraud" A typical scenario is once the organized airninal group changes the modes d operation, the existing programs are no longer effective n order to bring loss under control and to minimize it, a new strategy must be implemented This paper will discuss the basic concepts and illustrate some d the implementation techniques by utilizing SAS Enterprise Miner Since the fraud rate is relati\lely small (approximately 1 in 10,000 transactions) compared to millions d transaction per month, the model needs to be robust enough to yield high hit rate while keeps the false positive rate low f the latter is not low, major operation inconvenience and c:ustomer complaints will be resulted Based on the mode d operation, two major types d fraud, victimized and perpetrating, have been reported This paper wih iuustrate the model budding on the victimized fraud only A victimized fraud is defined as the crooks use several techniques such as making oounterfeit check, altering the check and dollar amount on a wrilten check or stealing the checks from customers and cash out from a victimized account Our strategy is to look for the deviation d fraudulent transaction behavior from normal customers' behavior pattern t will be good for preventing good customers from being victimized by fraud and also does not depend on knowing in advance what the fraud perpetrators are atlarnpting Sampling and Analysis File Preparation Nonnal Behavior Samples To build nonnal customers' transaction behavior pattem fa a representative set from the 11 million checking account database, first we took a small sample accounts from the 11 mhiion database The occurrence d business account frauds to consumer account frauds is close to 1:1 However 90 % d the checking accounts are consumers Therefore, a stratified sampling approach was used to sample equal number of business and consumer accounts A one-year worth of transactions were then extracted for the sampled accounts and several transaction parameters ~ clerived per-business cycle basis A business cycle is equivalent to one-month length wise except the cycle starting date and anding date vary from customer to customer n older to construct one observation per customer for predictive modeling, weighted average of 12 cycle for the transaction parameters were derived Number d transactions per cycle is used as the weight Fraud Behavior Samples Several thousands of fraudulent transactions occurred in 1996 & 1997 were included and the transaction parameters were computed in a similar way as normal behavior for each fraud transaction To be noted, majority d the frauds occurred within the same cycle due to short interval between transactions Data Analysis and Modeling SAS Enterprise Miner was employed to build the predictive models through various multivariate statistical approaches: neural networks, logistic regrassion and decision tree The diagram shown in Figure 1 mustrates an example d steps taken in predictive modeling The nput Data Source Node requires assignment d target and input variables for the model The distribution d each variable can be viewed by sifll)ly clicking on the variable and select "vidistribution" Figure 2 shows an example d detecting incorrect data from viewing distribution Time on book (ie age d the account) should be greater or equal to 0 n addition, percent of missing data for input variables can be lliewed so to help planning which data inputation strategy is needed Oecision Tree Analysis One primary advantage that tree modeling has over neural networks for our data is that tree-based models can handle missing values by treating them simply as another category d the analysis variable The entire 183

2 data (with 14,417 obselvalion) was used for the modeling and the partition between training and validation is 67% and 33% respectively The "root" of the tree is the entire data set The subsets form the "branches" of the tree Subsets that meet a stepping criterion and thus are not partitioned are "leaws" Any subset in the tree, including the root or leaves, is a "node" Figure 3 presents the results of tree analysis; the upper left summary table stores descriptive statistics about the number and proportion of observations correctly and incorrectly classified in the training and validation data sets; the upper right tree ring summarizes tree complexity, split balance and the discriminatory power of the tree; the right lower graph shows that the 16-leaf tree is optimal in terms of accuracy on validation data set Figure 4 shows the tree diagram for the fraud model The decision tree contains 12 nodes Two subsets (or leaves) were found to contain more than 75% of the fraud; the larger leave has 2143 observations and the small one has 12 observations The classification rate was 92% (1961 of 2141) for the bad accounts and 92% for the good ones That is, 8% of good accounts were misclassified as bad ones (false positive rate) Neural Netwolk Model and Logistic Regre&$ion An artificial neural network can be defined as a computer application that attempts to mimic the neurophysiology of the human brain in the sense that it learns from examples to find patterns in data By finding complex non-linear relationship in data, neural networks can help to make predictions about real-world problems The data was partitioned 50: 50 to training and validation respectively Several hidden layers and number of iterations were experimented Figure 4 presents a comparison of lift chart between neural network and logistic regre&$ion The horizontal axis gives the decile groupings d the predicted probability (the highest are on the left) The Wlrtical axis represents the actual fraud rate in each decile f the top 30% d the cases were targeted then the fraud rate woukl be 97% compared to the baseline fraud rate around 45% (with a lift value c1 22) Lift value is computed as cumulative response rate divided by baseline response rate The fluctuation represent random variation Frgure 5 presents a Lorenz curve (also known as ROC ane) which depicls the cumulative percent of aclual frauds that are captured in each cumulative decile t shows 90% ci all frauds would be captured, if the top 40% of the cases were targeted Overall, neural networks model outperforms logistic regression in terms ar % response rille and % captured response rate The overall misclassification rate is lowest in neural net model {75%), followed by decision tree model (83%) and logistic regression (17%) The two-way classification table ar the best neural net and decision tree models is presented in Table 1 CONCLUSON This paper ilustrates a successful example of use of predictive modeling identifying frauds in miftions of transactions Both the neural nelwo!k model and decision tree model achieved over 90% hit rate with less than 10% false positive rete Efforts in lowering false positive rate are und~ There are several keys to the success: a key strategy d looking for the deviation d fraud transaction behavior from normal one, generation of transaction-based parameters for the model, and utilization ar Enterprise Miner in data analysis and modeling n devalcping a major risk management mode~ the ease ar implementation and integration into the existing infrastructure are also the criteria to selection of an appropriate model Shiao-ping Lu 2669 Greer Road Palo Alto, CA splu@aimnetcom 184

3 Figure 1 Figure2 Percentage : : : me 185

4 186 Figure3

5 ' 0 (~5y4s Total \the ~e~-n, CA5Hirl otal ; Total i 11 io Total ,6'< , 00 < 141'9951 ' ' ~::i~~~~-"-' 1!14~ ~7\ 0 CUi\ 118 :lilt ?1& 1111 Total J the me~ n:, roramtj ~ j ~14\ 303\ % 697%' r::-::: , >= 2sL~s2l 557% 550\ < Figure 4 l ' tn_ _;_- ~ - ~ 81 '* ''rii o<o D :';";JYJ~\10'0 ilii O'< i!:f::";;,:;,'! ''~- ~ - g!rotal~ ; ;,2$',!~ t 0 "' the mean, MLEsJ J < >= ~ 230% 2871\o 0 770'0 713%

6 FigureS Figure6 '8} -ro :eo 50 #J - "_-,"-- '00 :~() ;t{} to /'' J /'' l/ '" 1/ 10 ~ ',,,~,;;;~~- ---~, '>: '< 7 ~ ' _:,, i'"' v ' : l/,: ' / V' v v v v / : ' : : :' ' ; '():~~:: '', 188

7 Table 1 Two-way classification table of Neural Network Model and Decision Tree Model Neural Network Model Frequency Actual Percent Row Percent 0 Fraud Good Fraud Good Decision Tree Model Frequency Actual Percent Row Percent 1 0 Fraud Good Fraud Good