Association Discovery. Janjao Mongkolnavin Department of Statistics Faculty of Commerce and Accountancy Chulalongkorn University

Transcription

1 Association Discovery Janjao Mongkolnavin Department of Statistics Faculty of Commerce and Accountancy Chulalongkorn University

2 Outline Market basket analysis Association rules Support-confidence framework Sequential analysis using association rules Dependent framework Understanding associations The cross-purchase skew methodology 2

3 Market Basket Analysis (1) 3

4 Market Basket Analysis (2) Market basket analysis uses the information about what customers purchase to provide insight into who they are and why they make certain purchases. Market basket analysis provides insight into the merchandise by telling us which products tend to be purchased together and which are most amenable to promotion. 4

5 Market Basket Analysis (3) This information can be used to suggest new store layouts, to determine which product to put on special, to indicate when to issue coupons. When this data tie to individual customers through a loyalty card or web registration, it becomes even more valuable. 5

6 Example of Output If customers buy product A they also tend to buy product B, x% of the time The word product can be extended to attributes,such as demographic, promotion, seasonal, and they can be extended to a group of products, such as all dairy products. Applications of basket analysis include cross-marketing, store layout, catalog design, product pricing and promotion. 6

7 Association Rules (1) Association rules represent patterns in the data without a specified target. 7

8 Association Rules (2) Although its roots are in analyzing pointof-sale transactions, association rules can be applied outside the retail industry such as Items purchased on a credit card Optional services purchased by telecommunication customers (call waiting, call forwarding) Banking services used by retail customers Unusual combinations of insurance claims 8

9 Building Association Rules (1) 9

10 Building Association Rules (2) Choose the right set of items Generate rules by deciphering the counts in the co-occurrence matrix Overcome the practical limits imposed by thousand or ten of thousand of items 10

11 Building Association Rules: Choose the Right Set of Items (1) 11

12 Building Association Rules: Choose the Right Set of Items (2) One way to compromise is to use more general items initially, then to repeat the rule generation on more specific items. The desired complexity of the rules also determines how specific or general the items should be. The appropriate level depends on its importance for producing actionable results eg. its frequency in the data. 12

13 Virtual Items Virtual items do not appear in the product hierarchy of the original items, because they cross product boundaries. Virtual items may include information about whether the purchased was made with cash or a credit card, the day of week or the time of the day the transaction occurred. 13

14 Anonymous - Identified For anonymous transactions, the only information is the date and time, the location of the store, the cashier, the items purchased, any coupons redeemed, and the amount of change. For identified transactions, we have more information on customers and their behavior over time such as demographic. These information can be incorporated into association rule by treated as virtual items. 14

15 Support-Confidence Framework The goal is to extract association of the form If A then B by using support and confidence. Support is define as the minimum percentage of transactions in the training set containing A and B. Confidence is defined as the minimum percentage of those baskets containing A that also contain B. 15

16 Support-Confidence Framework For example, suppose the training set contains 1,000,000 transactions (records), 10,000 of those transactions contain both A and B, the support of the association if A then B is 10,000 / 1,000,000 = 1%. If 50,000 of the transactions contain A and 10,000 out of those 50,000 also contain B, then the association if A then B has a confidence of 10,000 / 50,000 = 20% Thus confidence is the conditional probability of B given A ( P(B A) ). 16

17 Example When a customer buys a shirt, in 70% of cases, he or she will also buy a tie. We find this happen in 13.5% of all purchases. From this sentence, what is the percentage of support? What is the percentage of confidence? 17

18 Example Basketid A B C D E T T T T T T T T T T

19 Example The frequent sets for frequency threshold 0.4 are {A}, {B}, {C}, {D}, {AC}, and {BC}. Confidence for rule A -> C = 4 / 6 Confidence for rule B -> C = 5 / 5 19

20 Note on Confidence (1) Note that confidence alone does not distinguish whether the association between A and B is random or not. It has to be used together with other probabilities because we are interested in finding association between A and B where the occurrence of B is different from what would be expected by random chance. 20

21 Note on Confidence (2) For a statement When customers buy bread at a grocery store, 30% of the time they also buy milk, if 30% of all transactions in this store contain milk. Do you think this statement useful? 21

22 Note on Confidence (3) For example, it would be interesting to find that milk occurs 50% in all baskets containing coffee, which means there is a 20% lift of milk above the expected random occurrence of it. 22

23 Note on Support Generally, support is used to weight the significance of groups of transactions. Simple example will be we may ignore transactions with low support, or in other word, transactions that occur infrequently. However, sometimes, low-supported transactions can be useful in the discovery. 23

24 A Basic Algorithm 1. Create a set of one-dimensional columns whose frequency of occurrence are higher than the specified support. 2. Create a set of two-dimensional columns that are the combinations of the columns from (1) and test their support. 3. Repeat until there is no combination of columns satisfying the specified support. 24

25 What They Tell Us? (1) Rules based on high support and confidence factors represent a higher degree of relevance than rules with low support and confidence factors. However, if the support and confidence factors are too high, association rules may not be discovered. 25

26 What They Tell Us? (2) If the factors are too low, there will be many possible combinations of product associations, which is not what we want. 26

27 Sequential Analysis Using Association Rules Normally, association rules find things that happen at the same time. In order to use time series, the transaction data must have two additional features: A timestamp or sequencing information to determine when transactions occurred relative to each other Identifying information, such as account number or customer id that identifies different transactions as belonging to the same customer. 27

28 Sequential Analysis Using Association Rules: Procedure All items purchased by a customer are treated as a single order and each item retains the timestamp indicating when it was purchased. The process is the same for finding groups of items that appear together. To develop the rules, only rules where the items on the left-hand side were purchased before items on the right-hand side are considered. 28

29 Dependent Framework The Dependence Framework considers only transactions whose percent of cooccurrence is significantly different from random chance. In Statistics, the two events are independent if the probability of their cooccurrence is the same as the product of the probabilities of each event occurring individually. Eg. Probability(AB) = Probability(A) * Probability(B) 29

30 Dependent Framework: Example Both Coke and Pepsi occur separately in 50% of all baskets, and we expected that they should occur together 25% of all baskets by random chance, but the actual percent of cooccurrence was 0.01%. Thus, Expected(Coke + Pepsi together) = 25% Actual(Coke + Pepsi together) = 0.01% The difference between the two probabilities is large, so we assume that Coke and Pepsi are not independent. 30

31 Dependent Framework: Test the Difference In order to conclude whether two events are independent from each other, we need to test whether the difference between the expected co-occurrence and the actual co-occurrence are statistically significant. We can use the Chi-Squared test X [ ExpectedCoOccurrence ActualCoOccurrence = ExpectedCoOccurrence 2 2 ] 31

32 Dependent Framework: Test the Difference Note that the Chi-Squared test measures deviation from the expected random cooccurrence. It is equal zero if the deviation from random co-occurrence is zero, and it is greater than zero if the events are dependent. This statistic has a chi-square distribution. Thus, we can compare it against the Chi- Squared table and obtain a p value that will tell us the confidence level of our conclusion. 32

33 Beyond Statistical Dependence (1) We can better understand the dependence between two events by looking at its Impact and Lift. Impact is the ratio of the actual co-occurrence divided by the expected co-occurrence: Im pact = ActualCoOccurrence ExpectedCoOccurrence The impact is equal to one if the products are independent, and is different from one if the product are dependent. 33

34 Beyond Statistical Dependence (2) For example Impact(Coke+Pepsi) = 0.01/25 = which indicates that Coke and Pepsi are very dependent. Lift of event A on event B is defined as the difference between the expected and the actual co-occurrences of A and B divided by the frequency of A: [ ActualCoOccurrence ExpectedCoOccurrence] Lift( AonB) = FrequencyofOccurrenceofA 34

35 Beyond Statistical Dependence (3) Note that the value lift is between -1 and 1. The lift is close to zero if the items are independence. It is positive if the items are positively dependent. It is negative if the items are negatively dependent. 35

36 Beyond Statistical Dependence (4) For example: Life(Coke on Pepsi) = ( )/0.50 = This negative lift implies that Coke and Pepsi tend not to be in the same baskets. 36

37 Understanding Associations (1) When two events are negatively associated, they can either compete with each other in the same market segment, or they serve different market segments and they do not compete. 37

38 Understanding Associations (2) Likewise, two events can be positively associated because they complement each other or a third product complement them both. This distinct reasons can lead to very different marketing strategies. 38

39 The Cross-Purchase Skew Methodology (1) In this methodology, we further investigate the dependence association by using the third event (C) to pair with event A and B, and see if they show the same co-occurrence percentage. If they show the same co-occurrence percentage (no cross purchase skew), they confirm the assumption about the association between A and B. 39

40 The Cross-Purchase Skew Methodology (2) If they show different co-occurrence percentage (cross purchase skew), the association between A and B may have different explanation. 40

41 The Cross-Purchase Skew Methodology: Examples (1) Suppose that the orange juice and soda repel each other, leading to suspect that they may be substitutes. The investigation reviews that they exhibit cross-purchase skew when compared against whole-wheat bread. This reveals that A and B are not substitutes and do not compete. 41

42 The Cross-Purchase Skew Methodology: Examples (2) For example, It might be that healthy food, such as whole-wheat bread, tend to be purchased more often with orange juice than with soda. On the other hand, in the case of Coke and Pepsi, you may find little or no cross-purchase skew when pairing them with the third products. It may reveal that Coke and Pepsi are substitutes that compete in the same market. 42

43 Effective Basket Analysis Once groups of events that exhibit statistically significant non-random associations are found, should further analyze the cause of their non-random associations. 43

44 Q & A