TESTING WEB INTERVIEWING IN THE EGM MULTIMEDIA SURVEY EMRO Conference 2007 Carlos Lamas, AIMC EGM SAMPLE STRUCTURE Interview content Data collection Annual sample size Multimedia, household equipment, lifestyles and product consumption Only radio Face-to-face CAPI Telephone CAPI 43,000 38,000 Only newspapers Telephone CAPI 32,000 2 1
FIELDWORK PROBLEMS BECOME MORE SERIOUS General: decreasing response rates. Face-to-face Increasing costs for probability sampling designs. Lack of good interviewers (low wages in an expanding economy). Unsatisfactory quality of fieldwork. Telephone 23% of households are mobile-only. Costs of interviewing mobile-only individuals are three times higher than those to landline telephone households. The industry is reluctant to pay more simply for improving the fieldwork quality. 3 COULD WEB INTERVIEWING BE AN ALTERNATIVE? Internet usage penetration still low Total population Internet users...yesterday...last week...last month Frequency of usage...every day /Almost every day...several days per week...several days per month People 14+ (000) 37,439 8,838 12,852 14,377 8,713 12,305 14,396 (%) 100.0 23.6 34.3 38.4 23.3 32.9 38.5 4 2
INTERNET EVOLUTION IN SPAIN Penetration (%)) 45 40 35 30 25 20 15 10 5 0 1.0 1996 40.1 37.5 32.5 34.3 26.9 20.5 22.5 28.0 30.4 33.3 36.6 22.9 12.6 16.8 19.0 25.4 22.2 19.7 16.8 7.0 10.9 13.6 4.6 2.7 5.9 10.6 3.8 9.0 2.1 5.6 0.9 1.8 2.9 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 1 st 2007 Last 30 days Last week Yesterday 5 INTERNET USERS PROFILE (EGM 1st wave 2007) (%) 60 50 40 30 20 10 0 48,9 56,7 51,1 43,3 Male Female 14,4 13,1 19,1 28,2 19,4 22,4 15,3 14,3 12,5 19,6 14,3 13,1 21,5 7,2 6,9 6,1 1,5 7,0 41,4 46,0 30,7 16,6 7,9 1,6 14-19 20-24 25-34 35-44 45-54 55-64 65 or more Upper Upper middle Middle Lower Middle Lower Population Internet users (last week) 6 3
TEST STAGES Stage I : Questionnaire programming Stage II: Initial questionnaire test Stage III: Deciding questionnaire content Stage IV: Usage of incentives Stage V: Audience estimates SSI (Survey Sampling International) was selected for the questionnaire programming and as providers of the Access Panel for the sample selection. 7 Stage I: Questionnaire Programming January / June 2006 8 4
Questionnaire appearance 9 10 5
11 12 6
Stage II: Testing the questionnaire April 2006 Interview content: reduced questionnaire (all media excluding TV) No incentive, no reminder 930 invitations 181 completed interviews Response rate: 19.5% Interview duration (median value): 29.5 minutes 13 Stage III: Questionnaire content October 2006 Interview content: full EGM questionnaire (all media, household equipment, lifestyles and media consumption) Reminders, no incentive 1016 invitations 107 completed interviews Response rate: 10.5% Interview duration (median value): 64.5 minutes 14 7
Drop-out rate 100 90 98,4 80 70 60 81,1 74,6 74,2 72,5 69,7 (%) 50 58,6 40 47,5 47,1 45,5 45,5 44,7 44,7 43,9 30 20 10 0 Sociodemographics Dailies Sunday Suplements Magazines Cinema Radio TV Internet Outdoor Household equipment Product Consumption Lifestyles Final questions Completed interviews 15 Stage IV: Testing the usage of incentives November 2006 Interview content: reduced questionnaire (all media excluding TV) 2001 invitations 990: no incentive 1011: 5 euro incentive Differential response rates 25.0% with incentive 16.3% without incentive Interview duration (median value): 30.7 minutes 16 8
Stage V: Audience estimates December 2006 January 2007 Interview content: reduced questionnaire (all media excluding TV) 5 euro incentive, reminders 3.302 completed interviews Response rate : approx. 26% Interview duration (median value): 30.2 minutes Weights from the EGM Internet population (last week users) 17 NEWSPAPER READERSHIP AMONG INTERNET POPULATION (last week users) TOTAL DAILIES General information...pay...free Sports Economic / Financial Net AIR 128 131 123 202 149 260 Gross AIR 177 177 145 237 165 343 18 9
NEWSPAPER READERSHIP AMONG INTERNET POPULATION (last week users) TOTAL DAILIES General information...pay...free Sports Economic / Financial Net Total Readership 106 109 110 165 148 318 Gross Total Readership 159 153 133 190 170 335 19 SUNDAY SUPPLEMENT READERSHIP AMONG INTERNET POPULATION TOTAL SUPPLEMENTS Net AIR 128 Gross AIR 149 20 10
SUNDAY SUPPLEMENT READERSHIP AMONG INTERNET POPULATION TOTAL SUPPLEMENTS Net Total Readership 135 Gross Total Readership 170 21 MAGAZINE READERSHIP AMONG INTERNET POPULATION TOTAL MAGAZINES Weeklies Bi-weeklies Monthlies Net AIR 131 203 193 138 Gross AIR 227 236 191 226 22 11
MAGAZINE READERSHIP AMONG INTERNET POPULATION TOTAL MAGAZINES Weeklies Bi-weeklies Monthlies Net Total Readership 121 161 207 132 Gross Total Readership 245 230 210 254 23 INTERNET USERS YESTERDAY USERS FREQUENCY: Every day / Almost every day 137 141 24 12
CINEMA GOERS (LAST WEEK) CINEMA 171 25 RADIO LISTENERS AMONG INTERNET POPULATION TOTAL RADIO General content stations Specialized content (music, news, etc) stations Listeners per day 109 101 116 Average listening time 142 125 136 26 13
RADIO LISTENERS AMONG INTERNET POPULATION Listeners per week TOTAL RADIO General content stations Specialized content (music, news, etc) stations 103 95 107 27 MAIN CONCLUSIONS Obviously, the low penetration of the Internet means it is far too early to plan an eventual replacement of the present data collection systems. Web interviewing provides significantly higher audience estimates for the various media types. Potential weakness of the web samples is no doubt an important factor and it is true that the low response rate achieved could introduce additional bias. But I believe that the influence of the data collection methodology itself also plays an important role. Further research is needed to evaluate this assumption. 28 14