3 Ways to Improve Your Targeted Marketing with Analytics

Transcription

1 3 Ways to Improve Your Targeted Marketing with Analytics Introduction Targeted marketing is a simple concept, but a key element in a marketing strategy. The goal is to identify the potential customers who are most likely to buy what you are selling. Once you have identified this target group of people, you can focus your marketing efforts in their direction. This process could substantially decrease marketing costs. In this tutorial, you will use data from a Portuguese banking institution whose goal is to cut down on their telemarketing costs 1. They want to call only the clients most likely to subscribe to a long-term deposit. By using modern data mining approaches (TreeNet, MARS, and RandomForests), you will build a model to successfully identify which customers to target and explore the characteristics of this market. Tutorial 1) Open SPM : 1 [Moro et al., 2014] S. Moro, P. Cortez and P. Rita. A Data-Driven Approach to Predict the Success of Bank Telemarketing. Decision Support Systems, Elsevier, 62:22-31, June 2014

2 2) Click the folder shortcut to open a data file: 3) Locate bank.csv (or your chosen file name) and click Open.

3 The Activity Window, pictured below, will appear: This file contains 41,188 records of telemarketing calls at the Portuguese bank. On the left side of the Activity Window, you can see the variables in the data file. These include attributes of the prospect (i.e. age, marital status, education), economic indicators (i.e. consumer price index, employment numbers), and campaign variables (i.e. type of contact, date of last contact). At the bottom of the Activity Window, a row of buttons gives you options to view the data, create graphs, and run statistics, among other features.

4 4) At the bottom of the Activity Window, click the View Data button: The spreadsheet pictured below will appear with the data: Note that there are missing values and both categorical and numeric variables. Exact variable definitions can be found at

5 5) Return to the Activity Window via the shortcut : 6) Click Stats:

6 The Descriptive Stats dialog box will appear: 7) Check that Detailed Stats and Tables is selected along with all of the variables in the dataset. Enter 100,000 in the box next to Max. distinct values to track to be sure all values will be displayed. 8) Click OK.

7 The results will appear in a window with one variable per row: For example, you can see variable DEFAULT$ is 20.87% missing and has 2 distinct levels. 9) For more detailed statistics, click the Full button in the upper right corner of the window:

8 This view gives, for each variable, descriptive statistics, location variables, variability, quantiles, and frequency tables. Running descriptive statistics before building a model gives you a better idea about the variables you are including, or leaving out, of your model. 10) Scroll over until you reach the target variable, SUBSCRIBED. Click the plus sign next to Frequency Tables : Note that of the 41,188 records, only 11.27% of the calls ended in subscriptions.

9 11) Click the Model Setup shortcut : In the Analysis Method drop-down menu, choose TreeNet. As the Analysis Type, choose Logistic Binary. In the Variable Selection window, choose SUBSCRIBED in the Target column and the remaining variables in the Predictor column. Scroll through the list and UNCHECK the following variables: TEST80_20, TEST50_50, Y$, DURATION, and ID. These variables are not useful in the analysis.

10 12) Click the Testing tab: Typically, you would randomly partition the data into 80% to build the model and 20% to test the model. However, because we are using time-ordered data where many of the variables may be affected by time, you must use the first 80% of records to build the model and the last 20% to test. To do this, choose the TEST80_20 variable under Variable separates learn, test. This variable simply holds a value of 0 for the first 80% of records and a value of 1 for the remaining 20%.

11 13) Click the TreeNet tab: This tab holds the internal TreeNet controls, such as number of trees to build and how fast the trees learn from the data. The defaults are usually a good place to start and can be tweaked after you analyze the model. 14) Change the Optimal Logistic Model Selection Criterion to ROC area, but leave everything else as is. 15) Click Start.

12 The TreeNet Output window, pictured below, will appear once the model is built: The Output window shows a graph of the number of trees in the ensemble with its corresponding ROC value. The vertical green bar denotes the model with the optimal ROC: 9 trees at An ROC of 0.50 equates to random guessing so you ve extracted a good amount of information with this model. Typically, a TreeNet ensemble has more trees, but the engine only required 9 trees and including more would push into overfitting. 16) At the bottom of the Output window, click the Summary button:

13 The Summary Results window will appear with tabs on numerous details about the selected model: Again, you can see the ROC on the test data is Other reported performance measures include Lift and Misclassification Rate. 17) Click the Variable Importance tab:

14 This tab is a quick way to see which variables are the most important in predicting whether or not a potential client will subscribe to the bank s investment. Keep in mind, this model is completely up to you; if MONTH$ isn t an ideal predictor for learning about your potential customers, remove this variable and re-run the model. 18) Click the Prediction Success tab: The Prediction Success table lets you know how accurate your model was able to predict in each class. For example, this TreeNet model correctly predicted non-subscribers 95% of the time and subscribers 13% of the time. Ideally, you would like a higher accuracy in predicting subscribers; this is something you will keep in mind as you continue building the models in this tutorial. 19) Return to the Output window and click Create Plots:

15 The Create Plots dialog box will appear: 20) Check the box next to One variable dependence and Select All variables. Click Create Plots.

16 21) In the resulting window, click Show All to display the plots:

17 A partial dependency plot for each important predictor will be created:

18 Scroll through to see which values of the important variables affect the subscription the most. For example, prospects with a basic 4-year education or a university degree are more likely to subscribe. Using TreeNet for targeted marketing has improved random calling and given you an idea of how the predictors affect subscription. Next, you will see how a MARS model improves the performance of targeted customers while delivering output in a standard regression equation. 22) Re-open the Model Setup window : 23) Change the Analysis Method to MARS. Even though MARS is traditionally a non-linear regression engine, it can also be used for logistic binary cases such as this one. Check that the variables are still correctly selected from the previous run.

19 24) Click the Testing tab: To compare directly with the TreeNet model, keep the same testing method as Variable separates learn, test and select TEST80_20.

20 25) Click the Options and Limits tab: 26) Change the number of maximum basis functions to 60. This allows for more transformations in the predictors. 27) Click Start.

21 This output window shows you the number of basis functions in the model against the performance of the model. Because MARS is a regression engine, the MSE and R-squared values will still be reported, but can be ignored here. 28) Instead, open the Summary window to see the ROC performance: This model improved in targeting customers, with an ROC of 0.72.

22 29) Click the Variable Importance tab: The predictors have been used differently in building this model, so the importances have changed. 30) Click the Basis Functions tab: Here is where the logistic regression equation is laid out in terms of the basis functions (transformations of the predictors). Each basis function is described and the final model is listed at the bottom. This form of output is especially desired by those who are comfortable with standard regression.

23 31) Click the Prediction Success tab: With this MARS model, non-subscribers are correctly predicted 82% of the time and subscribers are correctly predicted 51% of the time. This is a more satisfactory result when considering which prospects you would like to reach. However, predicting subscribers around 50% of the time can be improved upon. You have now seen how MARS boosts performance and gives another option for viewing the model. Lastly, you will see how a Random Forests model gives an even better performance and shows us exactly who to call and what the cost/return will be.

24 32) Re-open the Model Setup window : 33) Switch the Analysis Method to RandomForests. Verify that classification Logistic Binary and the correct variables are still selected.

25 34) Click the Testing tab: By default, RandomForests uses out of bag data for testing. This is the records that are not sampled during the bootstrap process. However, we want to directly compare to our previous models and take the time-ordered records into account so you ll choose the same method.

26 35) Click the RandomForests tab: 36) For time-saving purposes, change the number of trees to build from 500 to 200. Additionally, change the number of predictors sampled to 5. As a rule of thumb, this number should roughly equal the square root of the number of predictors.

27 37) Click the Class Weights tab: We noticed from the MARS run that the model was doing a poor job of classifying subscriptions, which is the class of interest. For the RF model, we will tell the engine to upweight the smaller class (1) in an attempt to increase the prediction accuracy. The greater the class weight, the harder RF will work to classify that class correctly. The true class ratio is 1:8, but this may be too extreme so you will pick an intermediate ratio of 1:5. 38) Click Specify and change the class weight corresponding to class 1 to 5, as in the picture above. 39) Click Start.

28 The optimal model in an RF ensemble will always be the largest number of trees. 40) Open the Summary window: The RandomForests model achieved an ROC of 0.73 on the test sample.

29 41) Click the Variable Importance tab: Again, the variable importance scores have changed with the new model; many are hardly important at all. 42) Click the Prediction Success tab:

30 This table is where we see how well our model classified subscriptions under the new weighting scheme. In class 0, we have an accuracy of 57%. In class 1, we increased the accuracy to a satisfactory 80%. Other information in this table lets us know that this model resulted in calling only 54% of the total call list (8,484) while still reaching 80% of the subscribers. This could save the bank a substantial amount of money in telemarketing costs. Another interesting feature of the prediction success table is the ability to adjust the classification threshold. This allows the user to decide the optimal tradeoff between specificity and sensitivity. In this case, you would sacrifice accuracy in predicting subscribers in order to narrow down the telemarketer call list. 43) Use the up arrow to change the Threshold value to 0.370: This is just one example of adjusting the threshold. Now, the telemarketers are only calling half of the call list while still reaching 76% of subscribers. Once you are happy with a final model, you can score the model on a new data set. In this application, the bank would get a new list of potential customers, run the list of attributes through this model, and decide who to call based on likelihood of investing. 44) To run this process, return to the RF Output window and click Score:

31 A Score dialog box will appear: In a typical situation, you would select your new data file in the first field of this window. For illustration purposes, we will just generate predictions for the data set we have available (41,188 records) with our model. 45) Check the box next to Save Scores As and save the file as bank_scored.csv. Click Score.

32 The Score Results window will let you know how your model performed on the input data set. Overall, the ROC is Remember, this is much better than expected because we are scoring the data that was used to build the model. 46) Click the Prediction Success tab:

33 Here you can see the course of action for a much bigger list of prospects. If there were 41,188 new records, you would only call 24% of the full list to hopefully reach 70% of subscribers. 47) Open bank_scored.csv in Excel: The scored data file has a column for the predicted outcome of each potential client. This is the list you would use to let telemarketers know who to call.

34 You can also filter the predictions to only show contacts classified as subscribers: Conclusion TreeNet, MARS, and RandomForests are three different ways to improve a targeted marketing strategy. Each engine offers unique features to give insight to the problem at hand. Here, TreeNet grew a manageable ensemble that improved on the baseline model and provided partial dependency plots to determine predictor influence. MARS was able to further improve performance while delivering a traditional regression equation. Finally, RandomForests employed the power of class weights to give the best performance yet in correctly classifying subscribers. There are many more features in each engine that can optimize predictive accuracy; this simply serves as a quick example on a few ways to target market.