Research Design. The master plan to turn a conceptual research problem into a practicable empirical study. Requires a series of choices

Transcription

1 Research Design

2 Research Design The master plan to turn a conceptual research problem into a practicable empirical study. Requires a series of choices Exploratory, descriptive, causal Qualitative, Experimental, Survey If survey: Cross-sectional/longitudinal; single/multi-source Sample Measures Analyses

3 From Problem to Design: What s the best design for An advertising company wants to know what issues it needs to address in selling a new cleaning product. A food manufacturer has been given two sets of advertisement copy and want to know which one is better to produce sales, and if one is better for men than women. A high tech company has seen the turnover rate of its engineers rise in the past 12 months by 50%. Management wants to know if this is due to pay dissatisfaction, abusive managers, the way in which work is allocated or something else.

4 Exploratory & Descriptive Research Can be qualitative and/or quantitative Use primary (self-collected) or secondary (other-collected) data Exploratory: Loose rules & procedures, constant comparison, flexibility, iterative Descriptive: Tight structure, precise rules and categorization schemes, consistent procedures Present data in table form

5 Descriptive Table SMOKE Social Class Low Low-Middle High-Middle High Total YES NO Total 100% 100% 100% 100% 100% N =

6 Causal Designs: Covariation is not Cause Sales increase Price Reduction Yes No Yes 20% -- No 80% 100% 100% 100% Satisfied customer Stay close to customers Yes No Yes 30% -- No 70% 100% 100% 100% Does price reduction cause sales increase? Does staying close to customers cause customer satisfaction? Which came first? Reciprocal relations? Not related? X --> Y, Y <-- X, X <--> Y, X =\= Y Two variables: Four possible relationships. Six variables: 15 two-variable combinations or 4 15 = 1073 million potential relationships. Explore all or use theory to guide testing? Confounding factors? Conditioning factors? Effect often does not occur in presence of intervention. Design a study to ensure causality, rule out alternative explanations and identify moderators.

7 Classic Experiment: Training & Task Performance Control Group No training Pre-test units produced Post intervention round Post intervention round Difference by round Experimental Group Training Can we rule out alternative influences? Yes, if we randomize participants to groups, and measure performance blind to condition.

8 Validity Threats and Causality Internal validity: Are the results/effects obtained by the study really true Threats: History (specific events external to the study that generate the effect. In their absence, no effect would be observed. E.g., Announcement of price increase next month amplifies effect of a sale now) Test/Demand Effect (the experiment itself generates the effect) Hawthorne effect. Selection bias (randomization was incomplete/failed) External validity: Are the results/effects obtained by the study generalizable Threats: Laboratory studies with students may not apply in real world Require mundane realism (e.g., Erez s study of rudeness) Quasi-experiments in the field: Sacrifice some internal validity for more external validity

9 Survey Research Designs Use theory and experimental research to assume causality. Cross-sectional research: Collect all measures across time and assume causality of covariation is shown. Longitudinal research (allows for change/growth & trend analysis): Take measures of IV and DV at multiple times Demonstrate covariation works in one direction but not the other. Experience Sampling/Daily Diary. Moderation research: Show that relationship exists only in presence of some other factor, indicating that this factor turns on/off the effect.

10 Moderation to Suggest Effect Industry Manufacturing Hi Tech Small Large Small Large Total High 30% 60% 70% 40% 50% Innovativeness Low 70% 40% 30% 60% 50% n Relationship between size and innovativeness in overall sample: Not significant Take Industry into account, and we see that there is a link with large firms being most innovative in manufacturing and small firms being most innovative in high tech.

11 Other Design-Related Questions for Survey Research How to operationalize the concepts of interest? Are secondary data available or must we collect primary data? Secondary data often lack measurement depth (e.g., National Longitudinal Study of Youth) Sample Does representativeness matter? National probability sample Sample size: What n is need to get significant results? Survey length; frequency of data collection; mode of data collection (Smartphone?)