Mixed Mode Surveys in Business Research: A Natural Experiment. Dr Andrew Engeli March 14 th 2018

Transcription

1 Mixed Mode Surveys in Business Research: A Natural Experiment Dr Andrew Engeli March 14 th 2018

2 Structure of todays presentation The general context The natural experiment Resources Conclusion Coverage Sampling Response Measurement

3 Push to web as a (persistently) emerging mode The General Context

4 The uphill battle for surveys Increasingly embedded in all areas of life Survey fatigue The four challenges Coverage sampling nonresponse measurement The new normal: send a lot, get a few A cultural misfit is someone who answers the phone Multiple challengers

5 Push to web Long held view that online surveys are the future (e.g. Couper 2000) To date, online surveys still not replaced traditional telephone, F2F engagement In principle, address based sampling (ABS) provides greatest coverage Experiments have shown that mixed mode/partial online surveys: yield lower response rates for PTW versus telephone or postal PTW produces faster response rates that telephone or postal PTW has significant demographic biases

6 The promised land ONS (UK government) data collection transformation programme (DCTP) Push all surveys online by 2020 Cheaper Efficient Data joining with admin datasets What does this mean for the rest of us?

7 Push to web in the private sector: a natural experiment Wavehill s interest cheaper and more efficient sounds good, but what about our USP? Sink or swim? Two large scale business surveys in 2017 and 2018: Liverpool City Region Combined Authority Employer Skills Survey (1,856 completes) First Gloucester Countywide Skills Survey (899 completes)

8 CATI call CATI call Aberaeron Bristol Telephone team PTW team Emphasised telephone Emphasised online Complete interviews Send links + sweep LCRCA GFirst The mixed mixed modes

9 Obtaining sample 1. Coverage

10 Coverage challenges IDBR fabulous but obviously off limits HMRC fabulous but off limits Companies House extensive but poor data quality, limited variables CRM data great, but strong coverage bias Commercial sample Coverage is less good, bias random, data quality better

11 Do telephone and online yield equal samples? 2. Sampling

12 (IDBR Chi-square: p=0.68) (IDBR Chi-square: p=0.32) Survey demographics

13 Predicted probabilities by size

14 Predicted mode, controlling for other firmographics

15 Survey mode, comparing LCRCA and GFirst Logistic regression Number of obs = 2,589 LR chi2(14) = Prob > chi2 = Log likelihood = Pseudo R2 = mode Odds Ratio Std. Err. z P> z [95% Conf. Interval] size Micro Micro Small Small Medium Medium Large single markets Locally within an individual town or local area Nationally within England Regionally within the Liverpool City Region [if prompted lis Within the EU Within the UK dataset _cons

16 Mode comparison 3. Response

17 Survey completion rates Direct comparison of mode effects on response rates not possible for LCRCA 0.28 completes & 7.6 calls/hr, response rate 11.1% Gfirst: Telephone Push to Web Hours Number of interviews Engagement no of calls made Response rate 22.3% 30.1%* Completes per hour Calls per hour

18 Single item measures 4a. Measurement

19 Vacancies Survey dataset reported LCRCA GFirst Total None Vacancies 1, , Total 1, , Pearson chi2(1) = Pr = Vacancies Survey mode reported Telephone Online Total None Vacancies 1, , Total 1, , Pearson chi2(1) = Pr = Vacancies reported over 12 months

20 Survey dataset recruit LCRCA GFirst Total Recruited 1, ,806 Did not recruit Total 1, ,881 Pearson chi2(1) = Pr = Survey mode recruit Telephone Online Total Recruited 1, ,746 Did not recruit Total 1, ,815 Pearson chi2(1) = Pr = Recruitment in the last 12 months?

21 Hard to Fill Vacancies Survey dataset (HTV) LCRCA GFirst Total Yes No ,059 Total 1, ,842 Pearson chi2(1) = Pr = Hard to Fill Vacancies Survey mode (HTV) Telephone Online Total Yes No ,037 Total 1, ,782 Pearson chi2(1) = Pr = Hard to fill vacancies

22 Survey dataset train LCRCA GFirst Total Don't know No Yes 1, ,159 Total 1, ,749 Pearson chi2(2) = Pr = Survey mode train Telephone Online Total Don't know No Yes 1, ,093 Total 1, ,674 Pearson chi2(2) = Pr = Train

23 Survey mode, comparing LCRCA and GFirst Logistic regression Number of obs = 2,491 LR chi2(14) = Prob > chi2 = Log likelihood = Pseudo R2 = train Coef. Std. Err. z P> z [95% Conf. Interval] mode size Micro Micro Small Small Medium Medium (empty) Large markets Locally within an individual town or Nationally within England Regionally within the Liverpool City Within the EU Within the UK single Multi dataset _cons

24 Multiple response items 4b. Measurement

25 Multiple response variables

26 Survey mode Telephone Online Total Reasons for vacancies rr1 Staff turnover rr2 Business growth rr3 Internal promotion rr4 New skills required rr5 Other Total Cases Valid cases: 1778 Missing cases: 901 Overall Test(s) of Significance: Pearson chi2(20) = Pr = 0.239

27 Reasons for new skills demands

28 Survey mode Telephone Online Total chi2/p* inct1 New product inct2 New market Reasons for new skills demands inct3 New process inct4 Increased performance inct5 New technology inct6 Growth in existing markets inct7 Other Total Cases * Pearson chi2(1) / Bonferroni-adjusted p-values Valid cases: 1140 Missing cases: 1539 Overall Test(s) of Significance: Pearson chi2(62) = Pr = 0.002

29 Logistic regression Number of obs = 1,116 LR chi2(14) = Prob > chi2 = Log likelihood = Pseudo R2 = Logistic regression Number of obs = 1,116 LR chi2(14) = Prob > chi2 = Log likelihood = Pseudo R2 = inct4 Odds Ratio Std. Err. z P> z [95% Conf. Interval] mode size Micro Micro Small Small Medium Medium Large markets Locally wi Nationally Regionally Within the EU Within the UK single _cons New markets inct2 Odds Ratio Std. Err. z P> z [95% Conf. Interval] mode size Micro Micro Small Small Medium Medium Large markets Locally wi Nationally Regionally Within the EU Within the UK single _cons Increased performance Reasons for new skills

30 Survey mode Q137 Telephone Online Total Focus group recruitment Yes No , Total 1, , Pearson chi2(1) = Pr = 0.000

31 Focus group recruitment

32 Number of survey items showing significant Chi-square Firmographics 1 (9) Single response Multiple response Significant items Both LCRCA GFirst 11 (21) 1 (2) 4 (8) 34 (112) 14 (18) 8 (14) After treatment Significant items Both LCRCA GFirst Firmographics 0 (1) Single response 2 (11) 0 (1) 1 (4) Multiple response 5 (34) 0 (18) 1 (8) Total = 73 of 184 (39.6%) Total = 9 of 184 (4.9%) Summary

33 Costs and completion rates Resources

34 Survey costs

35 Push to web: a viable commercial strategy? Conclusions

36 The case for push to web For business surveys, push to web offer a number of advantages Good value for money, though not as much as hoped Will require ongoing measurement validation Uses different engagement skills, different kind of caller

37 Issues with push to web Can we trust fully automated followup? Will it work outside of targeted business populations? Individual level surveys: Different populations, different challenges We do not have the imprimatur of ONS/UK/Welsh Government Preference for mobile devices Studies show consistent demographic biases Wavehill conducting a pilot for National Institutes of Health Research in conjunction with University of East Anglia

38 Thank you For more information about Wavehill and the services that we provide please visit our website or follow us on Twitter: twitter.com/wavehilltweets

39 Survey mode Size band Telephone Online Total Micro Micro Small Size by mode Small Medium Medium Large Total 1, , Pearson chi2(6) = Pr = 0.277