TNM033 Data Mining Practical Final Project Deadline: 17 of January, 2011

Transcription

1 TNM033 Data Mining Practical Final Project Deadline: 17 of January, Develop Models for Customers Likely to Churn Churn is a term used to indicate a customer leaving the service of one company in favor of another company. The aim of this exercise is to develop models that predict whether a customer is likely to churn, i.e. the company would like to know how to characterize the customers who may soon churn. Moreover, based on the knowledge gained from analyzing and mining the data, you are required to propose a number of practical measures that the company can put in action to try to retain those customers likely to churn. The dataset to be used for this exercise, available from the course web page in the file churn.arff, has about 3333 customer records. Each customer is described by 20 attributes, plus a binary class attribute churn indicating whether the customer churned. A description of the meaning of each attribute is given in appendix. Your approach to the problem should follow the KDD process: data understanding (also called exploratory data analysis), data preprocessing, mining the data, and description of the interesting discovered patterns (knowledge). The suggested software to be used is Weka 3.6. However, feel free to try any ideas you may have to tackle the problem with any other software. Your final report must clearly document the following aspects. Describe your findings for each of the KDD steps mentioned above. Indicate how you parametrized each (Weka) algorithm you used in this exercise. Indicate performance measures for the models you have built and include a comparison between different models (see also point below). Refer which software tools you have used 1 and their purpose. Consider also the possibility of using graphs and images in your report since they can improve the clarity and the strength of your conclusions. 1 If a tool is available through a web page indicate its web address. 1

2 1.1 Data Understanding The first step in approaching a data mining problem is to delve into the data, identify any interesting relationships between the attributes, and formulate some initial hypothesis, i.e. possible associations between the attributes and the class. Graphical tools can aid you in this phase. You can find below a suggestion of some points to look at that may help you to better understand the data. 1. For each attribute find the following information. (a) The attribute type, e.g. nominal, ordinal, numeric. (b) Percentage of missing values in the data. (c) Max, min, mean, standard deviation. (d) The type of distribution that the numeric attribute seems to follow (e.g. normal). (e) Are there any records that have a value for the attribute that no other record has (i.e. unique values)? (f) Study the histogram of the attribute and note how it seems to influence the risk for churning. (g) Are there any outliers for the attribute under consideration? If you suspect of the existence of outliers for an attribute, you may consider the possibility of using box plots for outlier detection Observe whether the dataset has imbalanced class distribution. 3. Switch to the Visualize tab on the upper part of the screen in Weka to visualize 2D-scatter plots for each pair of attributes. (a) Does any pair of attributes seem to be correlated? (b) Which attributes seem to be the most/least linked to the risk for churning? Summarize in a table your findings concerning the predictive value of each attribute with respect to the churn attribute. (c) Investigate also possible multivariate associations of attributes with the class attribute, i.e. study scatter plots of two attributes X and Y and try to identify possible high(low)-churn areas (if any). 4. Are there any variables that can be eliminated? Justify your answer and motivate the possible benefits of doing so (if any). Compare your conclusions with the results obtained by using some attribute selection filters in Weka. Do not forget to indicate which filter(s) you have used and to give a brief description. 2 The course web page has links to some available easy-to-use tools that build box plots. 2

3 1.2 Data Preprocessing The second step is to preprocess the data such that the transformed data is in a more suitable form for the mining algorithms. You can find below some aspects you should consider. Obviously, you may consider other aspects of data preprocessing, such as creating new attributes from the existing ones. 1. Attribute selection. Select one or two subsets of attributes with good predicting capability and motivate your choices (recall your conclusions for points 3 and 4 above). 2. Handling missing values. 3. Eliminating outliers. 4. Discretization of numeric attributes. Do not forget to indicate which discretization method did you use, if discretization is used. 5. Normalization. Motivate the need of normalization (if needed at all). 1.3 Clustering Investigate the use of clustering techniques, for instance with K-means to segment the customers in order to get groups of customers with similar service usage characteristics. Profile the clusters, i.e. what can you learn about the types of records falling into each cluster (how can you describe in plain English each cluster). Note that this problem can be addressed by building classification rules (e.g. association rule mining can be used) that predicts the cluster attribute. If K-means is used, justify also the number of clusters you have chosen. Remember that measures such as the Sum of Square Errors (SSE) and cross-tabulating the clustering and class labelling (churn) can be useful to access the quality of the clusters Compare the clustering results with the two classes existing in the data, i.e. churners and non-churners. 1.4 Building Classification Models for the Data The third step is to use some classifier algorithms available in Weka to discover hidden patterns in the data. You should repeat the steps described below for each of the datasets you created during preprocessing. Include in your report a brief description of the classification algorithms you have used. 3

4 1. Start with OneR classifier. (a) What can you conclude? Compare your conclusions with your previous conclusions obtained in section 1.1. (b) Compare the accuracy of the classifier on the training set with the accuracy estimation obtained through 10 fold-cross validation. How do you explain the difference (if any)? 2. Use JRip classifier, i.e. the Weka version of the rule classifier RIPPER. (a) Build a classifier with and without rule pruning. Which one is preferable? Motivate your answer. (b) Summarize the patterns you obtained and compare with your previous conclusions. 3. Use J48 classifier, i.e. the Weka version of the decision tree classifier C4.5. Do not forget to append the tree obtained to your report. (a) Investigate the use of different J48 s parameters such as pruning and minimum number of records in the leaves. (b) Describe the patterns you obtained and compare with your previous conclusions. 4. Use of association rule mining (ARM) to build high confidence rules predicting churn. (a) Investigate the use of the Apriori algorithm. Remember that the implementation of Apriori algorithm available in Weka can only cope with discrete attributes. (b) Describe the patterns you obtained and compare with your previous conclusions Models Performance In the previous step, you have built several models. You need to assess the quality of the models and compare the different models. 1. Weka outputs several performance measures. Choose some of the performance measures and motivate your choice. 2. Summarize in a table the performance measures for each classifier and each dataset. 3. What can you conclude? 4

5 1.5 Conclusions Summarize your major findings and describe which risk factors for churning have you found in the data, after analyzing it. Based on your findings, indicate also some practical measures the company can put in action to retain those customers who are likely to churn. This last aspect is after all what the company is interested in. Lycka till! 5

6 Appendix You can find below a brief description of the meaning of each attribute. 1. State: discrete variable that indicates the state where the customer lives. 2. Account Length: integer variable that indicates how long the account has been active. 3. Area Code. 4. Phone Number. 5. Inter Plan: binary variable indicating whether the customer has an international plan. 6. Voic Plan: binary variable indicating whether the customer has a voice mail plan. 7. No of Vmail Mesgs: integer variable that indicates the number of voice mail messages. 8. Total Day Min: continuous variable that indicates number of minutes the customer used the service during day time. 9. Total Day Calls: integer variable that indicates the number of calls during day time. 10. Total Day Charge: continuous variable that indicates how much was charged for using the service during day time. 11. Total Evening Min: continuous variable that indicates number of minutes the customer used the service during evening time 12. Total Evening Calls: integer variable that indicates the number of calls during evening time. 13. Total Evening Charge: continuous variable that indicates how much was charged for using the service during evening time. 14. Total Night Min: continuous variable that indicates number of minutes the customer used the service during night time. 15. Total Night Calls: integer variable that indicates the number of calls during night time. 16. Total Night Charge: continuous variable that indicates how much was charged for using the service during night time. 17. Total Int Min: continuous variable that indicates number of minutes the customer used the service to make international calls. 6

7 18. Total Int Calls: integer variable that indicates the number of international calls. 19. Total Int Charge: continuous variable that indicates how much was charged for international calls. 20. No of Calls Customer Service: integer variable that indicates the number of calls to customer support service. 21. Churn: binary class variable. 7