Report for PAKDD 2007 Data Mining Competition

Transcription

1 Report for PAKDD 2007 Data Mining Competition Li Guoliang School of Computing, National University of Singapore April, 2007 Abstract The task in PAKDD 2007 data mining competition is a cross-selling business problem from consumer finance industry. The objective is to predict the propensity of the credit card customers to apply for a home loan. The training data contains cases, 40 features, and 1 class label variables. After data pre-processing, we identified significant features with information-gain-based feature selection method, such as the number of Bureau enquiries for mortgages and loans, age, current residence months, and etc. We selected the top 14 features and applied logistic regression method to the training data. Compared with data mining methods, the results showed that the logistic regression method achieves the best AUC score from 10 fold cross validation. With the mentioned information, we built our final classifier with the whole training data. The AUC score is about We expect the similar performance on the withheld data. 1 Introduction PAKDD 2007 Data Mining Competition [1] is one of the events in the 11th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD 2007). The background of the competition is that a financial institution has two groups of customers: one with credit cards and another with home loans. The institution would like to identify which customers are likely to apply for a home loan in 12 months after they apply for a credit card. The data for this task is the available information when the customers apply for a credit card. The main task in the competition is to predict the propensity of the credit card customers, who do not have a home loan, to apply for a home loan. 2 Data summary There are cases in the provided data, in which have the known class labels and 8000 hold out the class labels as the hidden test cases. Among the cases with known class labels, 700 customers have applied a home loan in 12 months after the credit card application. We called these 700 cases as positive data in this report. The information is summarized in Table 1. In the data set, there are forty-one features, excluding the customer IDs which cannot be used in the prediction and model building. The target variable, or the class label, indicates whether the customers have applied home loans. The other features are demographic data, credit information, and other related data. One feature, B_DEF_UNPD_L12M, only has (known) value 0. This feature is removed from the data before further analysis.

2 In the remaining data, including those for training and withhold, there are missing values (excluding the withheld class labels). The number of features with missing values is 9 for the entire data. The names of the features with missing values are listed in Table. The feature DISP_INCOME_CODE is the one with the most missing values it has missing values (about 71%). known Table 1 Summary of the available data Class label Number of cases Positive 700 negative Held-out 8000 Total Table 2 Names of features with missing values CHQ_ACCT_IND AMEX_CARD DINERS_CARD VISA_CARD MASTERCARD RETAIL_CARDS DISP_INCOME_CODE CUSTOMER_SEGMENT CREDIT_CARD_TYPE ANNUAL_INCOME_RANGE 3 Experiment In this work, we tried to build models to predict the class labels in the withheld data set. A data mining package Weka [4] is used as the main tool for data preprocessing, feature selection, model building and prediction. For the purpose to verify which data mining method is better for this application, the training data is randomly split into 10 fold for cross-validation. 3.1 Invalid values of nominal features In the data, some nominal features have invalid values. For example, feature MASTERCARD has invalid values of 1 and 2. For these invalid values, we replaced them with the mode of the corresponding features. 3.2 Discretization To apply certain data mining methods, we need to discretize the numeric data into nominal data. The discretization method we used is the method proposed by Fayyad and Irani [2]. This method selects a cutting point in the current data set with maximum information gain and split the data set into two subsets. Then the method recursively applies to each subset, until the Minimum-Description-Length-based stop criterion is reached.

3 3.3 Missing value processing Some data mining methods can deal with missing values, while others methods cannot. In this work, the missing values are replaced with the mean for the numeric features and the mode for the nominal features. 3.4 Feature selection In our experiment, we have tried different feature selection methods: chi-square, information gain, and gain ratio. These three feature selection methods measure the strengths of each feature for classification with the criteria in their names, and rank the features based on the measures. The top 14 features from these four methods are listed in Table. Table 3 Top 14 selected features Order Information Gain 1 B_ENQ_L6M_GR3 2 B_ENQ_L12M_GR3 3 CURR_RES_MTHS 4 AGE_AT_APPLICATION 5 PREV_RES_MTHS 6 B_ENQ_L3M 7 B_ENQ_L12M_GR2 8 B_ENQ_L6M 9 B_ENQ_L1M 10 RENT_BUY_CODE 11 MARITAL_STATUS 12 ANNUAL_INCOME_RANGE 13 A_DISTRICT_APPLICANT 14 SAV_ACCT_IND 3.5 Data mining methods applied In our experiment, we tried different data mining methods. After comparison, we selected logistic regression as our final method. Logistic regression [3] is a non-linear transformation of the linear regression, which applies to the case when the dependent variables are discrete. 3.6 Summary of experiment The data preprocessing and model building process is summarized in Figure 1. The main evaluation measures we used are ROC and AUC (ROC refers to receiver operating characteristic and AUC refers to the Area under the curve). Our results show that logistic regression gives the best AUC score. Figure 2 shows the ROC curve from the training data.

4 Value merging discretization Missing value replacement Feature selection Model building Figure 1 The procedure to process the data Figure 1 ROC curve from logistic regression model 4 Discussion After analyzing the data, we have the following insights from PAKDD 2007 competition data: 1. The most significant features for this cross-selling problem are: B_ENQ_L6M_GR3, B_ENQ_L12M_GR3, and other Bureau enquiry information. These features show the customers interests in mortgages. The customers may

5 have contacted with different financial institutions, and the contacted financial institutions did more enquiries from the Bureau. 2. The residence history is an important indicator for a home loan. If the customers rent a flat, it is more probable for them to apply for a home loan. If they have stay in a long time, it is probable for them to switch to a new home. 3. The age and marital status affect the home load application. These are the main factors to indicate whether the customers need a flat. 4. The income level is another main factor for a home loan. It indicates whether the customers can afford a home loan. Reference: [1] PAKDD 2007 Data Mining Competition, in: (2007). [2] U.M. Fayyad, K.B. Irani, Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning, in: Proceedings of 13th International Joint Conference on Artificial Intelligence (IJCAI) (1993) [3] D.W. Hosmer, S. Lemeshow, Applied logistic regression (John Wiley & Sons, New York, 2000). [4] I.H. Witten, E. Frank, Data mining: practical machine learning tools and techniques with Java implementations (Morgan Kaufmann, San Francisco, 1999).