Experimental Design: Part I. Overview 1/31/11. MAR 6648: Marke=ng Research February 3, 2010

Transcription

1 Experimental Design: Part I MAR 6648: Marke=ng Research February 3, 2010 Overview What are the basic features of an experiment? How do those features get implemented in a real experiment? How do we adapt experiments to meet our goals and resources? 1. Experimenta-on is the conscious manipula-on of one or more variables by the experimenter in such a way that its effect on one or more variables can be measured. 2. The variable being manipulated is called the independent variable (a.k.a. cause). 3. The variable being measured is called the dependent variable (a.k.a. effect). 4. Elimina-on of other possible causal factors: i.e., the research design should rule out the other factors (exogenous variables) as poten-ally causal ones. 5. This is typically done through random assignment to condi-on 1

2 An example of an experiment Suppose you want to know whether commercials make people enjoy TV shows less This means you ll want to have some shows without commercials, and some shows with them Therefore, commercials (or not) is the independent variable And you ll want to measure enjoyment of the TV shows they watch Therefore, enjoyment is the dependent variable Condi=ons Not in terms of what you can and can t do Each independent variable (or combina=on of IVs) is called a condi=on Condi=on 1 Condi=on 2 Hypotheses Experimenta=on is essen=ally the process of trying to determine which of two hypotheses is not false The null hypothesis: H 0 : Usually that there are no differences between condi=ons The alterna=ve hypothesis: H 1 : usually that there is a difference between condi=ons P- values in stats essen=ally represent the likelihood that we found evidence for H 1 by chance alone 2

3 Confirma=on Bias We are inclined to confirm our beliefs but less inclined (or able) to disconfirm them A real world example: Business managers don t keep track of those they don t hire Why? Theories lead to unwarranted confidence Inability to search out disconfirma=on Fixa=on or mental set Control condi=on Control condi=ons allow us to see that our manipula=ons caused (or didn t cause) a change in the dependent variable Usually a control condi=on is just no manipula=on This is some=mes done by adjus=ng when you run your manipula=on Some=mes, though, you want to compare your new manipula=on to what s typically done now The control condi=on may be the standard or default Random Assignment This essen=ally means that any one par=cipant is equally likely to be in any condi=on Usually you put your condi=ons in random order, and assign par=cipants in the order that they arrive Computers now allow you to assign people on the spot Randomizer.org or random.org are good sources 3

4 An example of an experiment The hallmark of an experiment is random assignment to condi=ons Let s say the groups (the commercial watchers and the people who watch it straight through) now look different! Random assignment means that the two groups should not have differed systema=cally at the start It also means that only your independent variable was different between groups Random assignment and manipula=on of the IV mean that you can infer that the IV causes a change in the DV An example of an experiment Ques=on: do commercials make you enjoy a TV show less? Do people correctly predict this? Randomly assign your par=cipants to groups Half will predict how they enjoy a TV show with or without them, half will actually experience it and report how they feel Half will watch a TV show with commercials, half will watch the same show without them Measure enjoyment or predicted enjoyment An example of an experiment Preference for show Forecasters Watchers Con=nuous Interrupted Nelson, Meyvis, & Galak,

5 Example Objec=ve: GAP wishes to gauge whether new more aggressive sales techniques employed by store assistants increase sales What is the best experimental design? Experiment 1 Design: 50 stores are sampled at random and assistants are trained in the new approach Metric = Average sales for the 50 stores in the next six months MINUS Average sales for the 50 stores in the prior six months Nota=on X = Exposure of a sample to the independent variable (i.e., what we manipulate treatment ) O = Observa=on of measurement of the dependent variable (i.e., what we measure / want to affect) Movement through =me is represented by the horizontal arrangement of Xs and Os from len to right. 5

6 Experiment 1: One group before aner Causal Effect of X = O 2 - O 1 Problems with this design? History or matura=on Defensiveness Mortality Instrumenta=on Experiment 2 Design: 50 stores are sampled at random and the assistants are trained in the new approach Another 50 stores are sampled at random as control Metric = Average sales for the 50 test stores in the next six months MINUS Average sales for the 50 control stores in the next six months 6

7 Experiment 2: Two group only aner Causal Effect of X = O 2 - O 1 Problems with this design? Experiment 3 Design: 50 stores are sampled at random and assistants are trained in the new approach Another 50 stores are sampled at random as control Metric = Average sales for the 50 test stores in the next six months Average sales for the MINU 50 test stores in the prior six months S MIN Average sales for the 50 control stores in the next US six months Average sales for the MINU 50 control stores in the prior six months S 7

8 Experiment 3: Two group before aner Causal Effect of X = O 4 - O 3 (O 2 - O 1 ) More Advanced Experiments We have so far mainly looked at simple experiments But onen we need to test several variables When deciding on a marke=ng plan for a new product there are many factors involved Factorial Design Suppose we wish to test both product price and web- design for an e- business Design Design 1 Design 2 Price $9.99 $14.99 $19.99 Full Factorial Design! 8

9 Interac=ons and main effects Factorial Design What do we do if we have many factors and levels? Example: 5 prices, 4 product designs, 3 ad- copies 5*4*3 = 60 experimental cells! Solu=on: Use a frac=onal factorial design Only use a subset of all 60 cells in experiment Rely on regression analysis to extrapolate La=n Squares 1 st ad 2 nd ad 3 rd ad 4 th ad Group #1 Group #2 Group #3 Group #4 9

10 La=n Squares 1 st ad 2 nd ad 3 rd ad 4 th ad Group #1 α β δ γ Group #2 β γ α δ Group #3 γ δ β α Group #4 δ α γ β An experiment? Steve was interested to see how much labels on wine bovles affect how much people enjoy the wine inside them. At a party, he served the wines like normal, leaving the bovles out for people to pour from, labels s=ll on. He asked everyone to indicate which wine they liked the best. At the next party he threw, he poured the wine into decanters, so that his guests couldn t see the labels when they poured the wine. They again indicate which wine they liked best, and they had different preferences from the last party. An experiment? The owner of two McDonalds franchises here in Gainesville wants to see if transac=ons run more quickly if he uses both drive- thru windows or only one. He picks one restaurant to use both windows at all =mes for a month, and the other he has closed at all =mes for a month. He finds that the drive- thru that uses both windows has notably faster service =mes. 10

11 An experiment? An experiment? Summary Experiments are very useful for determining causality The main hallmarks of experiments are random assignment to condi=on, manipula=on of the independent variable, and a control group There are many different types of experiments, which vary largely on whether they are run within or between subjects (or both), when the manipula=on is run, and how many condi=ons are used 11