Death by Survey Length

Transcription

1 1 Death by Survey Length

2 2 The art of prophecy is very difficult, especially with respect to the future. Mark Twain

3 Mr. President, how long to write a speech? 'That depends. If I am to speak 10 minutes, I need a week for preparation. If 15 minutes, 3 days. If half hour, two days. If an hour, I am ready now.' 3

4 PANELIST COMPLAINTS ABOUT SURVEYS The survey was WAAAAY too long and there was too much clicking required. I hate the multiple tables/grids. I don't like having to pick an answer when what I want to say is not applicable. Painfully long. The survey did not allow me to say I don t use the product so my answers were not based on any knowledge. I can t stand surveys with buttons across and all those rows down. They give me a headache. Not ask the same questions over and over and over and over. ONCE is plenty. My answers do not change. 4 It s hard for me to rate/answer questions about a store I only go to once in 6 months or so.

5 TODAY S LEARNING OBJECTIVES Understand the negative correlation between survey length and data quality Understand the implications of declining response rates on sample and data capture Leave with some suggestions on how to start making changes (internally and externally) 5

6 YOUR Senior Research Manager, Unilever Canada The Senior Vice President, AskingCanadians The President, Millward Brown Canada The Ugly 6

7 7 An Agency Perspective

8 THE STANDOFF 8

9 BEWARE OF BONUS KILLERS Improvements of 9% 23% to in aided awareness for large brands when we moved from a to minute survey 9

10 Seconds THE HIGHWAY TO HELL TIME TAKEN TO ANSWER QUESTIONS DECREASES WITH SURVEY LENGTH Source: Lightspeed GMI

11 GUITARS & DEMOCRATS What made you decide to purchase a Fender Strat? Jimi Hendrix played one Jimi Hendrix played one My brand Is modern Is contemporary Is up to date Keeps up with the times 11

12 POP QUIZ: POINT IS A SURVEY CONSIDERED TOO LONG Anything more than a few minutes More than 10 Minutes More than 15 Minutes More than 20 Minutes More than 30 Minutes At what point do you think a survey is too long? Anything more than a few minutes N=12,556 Cumulative % 7% 7% More than 10 Minutes 25% 32% More than 15 Minutes 24% 56% More than 20 Minutes 21% 78% More than 30 Minutes 17% 95% Never Too Long 5% 100% 12 LSR Quick Poll Survey July, 2012 (i.e. asked among panelists those willing to take surveys) At what point do you think a survey is too long?

13 THIS WILL CHANGE WITH OR WITHOUT US People are growing intolerant of long surveys RESPONDENTS WHO BELIEVE OVER 15 MINS too long 43% PC/LAPTOP 70% SMARTPHONE Main reasons respondents decide not to finish a survey Survey too long Repetitive questions Not interesting topic Incentive too low 31% 21% 43% 61% Survey Length is a problem across all methodologies, but so is reciprocity and respondent engagement 13

14 MANIFESTO FOR CHANGE Try your own survey, and reflect on the experience. Hard. Respect respondents, and make <15mins the goal (for now). Format and time for mobile, and whatever is next. Experiment with techniques like data imputation and adaptive brand lists. Be honest with your clients about the risks to data quality (more is less). Stand together. 14

15 15 A Panel Perspective

16 DESKTOP MOBILE MOBILE VS. DESKTOP PENETRATION RATES Mobile vs. Desktop Penetration Among AskingCanadians Panellists May 14 Jun 14 Jul 14 Aug 14 Sep 14 Oct 14 Nov 14 Dec 14 Jan 15 Feb 15 Mar 15 Apr 15 19% 20% 21% 22% 23% 22% 24% 24% 25% 25% 26% 26% 81% 80% 79% 78% 77% 78% 76% 76% 75% 75% 74% 74% 16

17 DESKTOP ANDROID iphone ipad MOBILE DEVICE RATES Average Response and Completion Rates by Device & Length of Interview Device Rates 1 5 Min 6 10 Min Min Min Min Min 40 Min+ RR 28% 23% 19% 17% 14% 10% 6% CR 85% 72% 70% 52% 50% 46% 39% RR 17% 16% 12% 11% 9% 6% 2% CR 69% 70% 64% 42% 47% 41% 32% RR 20% 18% 15% 15% 11% 7% 4% CR 74% 73% 67% 49% 56% 50% 44% RR 34% 30% 26% 25% 23% 18% 14% CR 85% 89% 85% 83% 74% 70% 65% 17

18 DATA QUALITY OPEN END QUESTIONS Open-end lengths, on average, are 150% longer on mobile devices vs desktop for 20 minute surveys or shorter Open-end usable data (i.e. user types in asdf, *^^&, etc..) is 40% greater for 20 minute surveys or shorter Rates become equal at 30 minutes or greater 18

19 CLIENT READINESS Average Length of Interview on AskingCanadians Panel Month Apr-14 May-14 Jun-14 Jul-14 Aug-14 Sep-14 Oct-14 Nov-14 Dec-14 Jan-15 Feb-15 Mar-15 Apr-15 LOI 16 minutes 16 minutes 17 minutes 18 minutes 20 minutes 20 minutes 20 minutes 20 minutes 22 minutes 23 minutes 25 minutes 25 minutes 25 minutes Percentage of Screenouts Due to Mobile Sniffout/Screenouts Month Screenout % Jan % Feb % Mar % Apr % in One Year 19

20 IF CHANGE ISN T ADOPTED Online panels will run out of supply, in their current form Panel companies are starting to/will refuse to work with clients who aren t adapting Data collection pricing will increase, substantially 20

21 BEST PRACTICES Ask what you need, not what you want U/X needs to be part of survey creation; data is only as good as design Think of respondents as people, not data 21

22 22 A Client Perspective

23 BALANCING ACT Business Partner Research Partner Align on business priorities Deliver actionable insights ROI Dynamic resource allocation Learn as much as we can 23

24 Less is more Time is money Higher engagement and completion rate 24

25 The risk to change now is too high! New surveys and tools lose the ability to analyze trends over time Some surveys are too short and critical details go missing There is often pressure to increase the number of questions in a survey to ensure it addresses all relevant issues 25

26 TIME ISN T EVERYTHING Shorter surveys may be just one piece of the puzzle Tighter and more focused objectives Identify the best technique / methodology We need to embrace the disruptive change digital technology is bringing to our Industry Be better at collecting and organizing insights Learn from it vs. defending against it 26

27 27 WRAP UP

28 KEY SUMMARY POINTS Research agencies are seeing a negative impact of questionnaire length on data quality, and many of are recommending to move to shorter device agnostic surveys immediately. The user experience isn t a commodity. Researchers need to treat respondents as a people to ensure the sustainability of online panels. When we are no longer able to change a situation, we are challenged to change ourselves Victor Frankl We need new capabilities that allow more insights balancing simplicity with robustness. 28

29 29 Questions?