Multi-mode survey administration: Does offering multiple modes at once depress response rates?

Transcription

1 Multi-mode survey administration: Does offering multiple modes at once depress response rates? Jocelyn Newsome, Kerry Levin, and Pat Dean Brick, Westat Pat Langetieg, Melissa Vigil, and Michael Sebastiani, IRS Office of Research* *The views expressed herein are those of the authors and do not represent those of the IRS. AAPOR Conference, Boston May 16, 2013

2 Background IRS Individual Taxpayer Burden (ITB) Survey Measures taxpayer burden in terms of time and money spent on: Recordkeeping Tax planning Gathering tax materials & software Professional tax help Completing & submitting tax returns Results used as input for Taxpayer Burden Model IRS draws sample of 20,000 taxpayers* *For this study, we used a large subset of comparable data, since data collection is ongoing. 2

3 2010 ITB Survey Experiment Paper Survey versus Sequential Multi-mode 75% Paper survey 25% Web Survey 3

4 2010 ITB Survey Experiment Results Paper Survey Sequential Multi-Mode 42.3% 39.4% overall response rate overall response rate n=12,509 n=4,210 4

5 2010 ITB Survey Experiment Results Paper Survey Sequential Multi-Mode 42.3% 39.4% overall response rate 627 Web survey completes n=12, Paper survey completes 5

6 2010 ITB Survey Experiment Results 42.3% 39.4% 14.90% Web Complete Paper Complete 42.00% 24.40% Paper Sequential Multi-Mode n = 16,719 6

7 2011 ITB Survey Design Contact 1: IRS Prenote Contact 2: Paper survey package with web instructions Contact 3: Thank you/ reminder postcard Contact 4: Paper survey package with web instructions Contact 5: Thank you/ reminder phone call or postcard Contact 6: Paper survey package with web instructions 7

8 Mixed-Mode Designs Sequential Modes Concurrent Modes Traditionally, mixed mode surveys have offered different modes sequentially. Most studies have found that offering respondents a choice of modes depresses response rates. Messer and Dillman (2011) Web-plus-mail sequential design gave a respectable response rate with a high proportion responding by Web. Mail mode outperformed push to the Web. Griffin, Fischer, and Morgan (2001) Found that offering multiple modes of response in a mailing led to a drop in overall response. Gentry (2009) Offering respondents a choice between a paper diary and an on-line ediary depressed the response rate for the choice group by 4.2 percentage points. Medway and Fulton (2012) Meta-analysis of concurrent vs sequential Web option in a mail survey found that including a concurrent Web option significantly lowered response rates. 8

9 Paradox of Mode Choice Allowing respondents to choose how they complete the survey makes it more likely they won t complete the survey at all. Possible Explanations Medway and Fulton (2012) Those who intend to complete on the Web experience some technical difficulties. Some respondents may decide to use the Web option, but never actually do the Web survey. Simply being asked to make a choice entails an additional cognitive burden that functions to deter response. 9

10 2011 ITB Survey Design Contact 1: IRS Prenote Contact 2: Paper survey package with web instructions Contact 3: Thank you/ reminder postcard Contact 4: Paper survey package with web instructions Contact 5: Thank you/ reminder phone call or postcard Contact 6: Paper survey package with web instructions 10

11 11

12 12

13 Impact on Response Rates 2010 Survey Sequential mode offering 2011 Survey Concurrent Mode Offering 43.4% 49.6% overall response rate* overall response rate *Overall response rate for respondents assigned to the sequential mode treatment who also received an incentive. All 2011 respondents received an incentive. In the 2010 survey, the highest response rate was among those who were assigned to the paper survey with an incentive: 47.6%. 13

14 Impact on Web Completes 2010 Survey Sequential mode offering 2011 Survey Concurrent Mode Offering 38.3% 5.5% of those offered web first completed by web of sample completed by web* *Although this is a lower web completion rate than in the 2010 survey, it is necessarily higher than it would have been if a paperonly methodology was adopted. It is possible that some of these respondents would not have completed a paper survey and therefore it is possible (although not certain without further analysis) that this methodology reduced non-response bias. 14

15 Conclusions It appears to be possible to offer two modes concurrently without adversely impacting response rates, if it is designed carefully. In order to avoid the paradox of choice, avoid emphasizing to respondents that they must choose a mode. You have the choice of completing this survey by mail OR online. VERSUS Please complete the enclosed survey. You can also complete it online. 15

16 References De Leeuw, E. (2005). To mix or not to mix data collection modes in surveys, Journal of Official Statistics, 21, Fricker, R. and Schonlau, M. (2002). Advantages and disadvantages of internet research surveys: Evidence from the literature, Field Methods, 14, Gentry, R. (2009). Offering Respondents a Choice of Survey Mode: Use Patterns of an Internet Response Option in a Mail Survey, Paper presented at DC AAPOR Web Survey Methods Workshop, September 10, Griffin, D., Fisher, D., Morgan, M. (2001). Testing an Internet Response Option for the American Community Survey, Paper presented at the American Association for Public Opinion Research, New Orleans, May Medway, R. and Fulton, J. (2012). When more gets you less: A meta-analysis of the effect of concurrent web options on mail survey response rates. Public Opinion Quarterly, 76, Messer, B. and Dillman, D. (2011). Surveying the general public over the Internet using address-based sampling and mail contact procedures, Public Opinion Quarterly, 75, Olson, K., Smyth, J., Wood, H. (2012). Does giving people their preferred survey mode actually increase survey participation rates? An experimental Examination, Public Opinion Quarterly, 76,