Found data. Administrative data. End users system (FTP, XML) Transaction data from private sector Published prices Big/organic/unconventional data

Transcription

1 A short recap 1

2 Found data Administrative data Reusing data from other official/government institutions End users system (FTP, XML) Transaction data from private sector Published prices Big/organic/unconventional data 2

3 Data collection Modes: Face to face Paper and pencil Telephone Web 3

4 Total survey error framework (Groves et al. 2004, Fig. 2.5)

5 Cognitive processes in Answering Questions A simple model Comprehension of the question Retrieval of information Judgement and estimation Reporting an answer Source: Tourangeau

6 Sampling and Nonsampling errors Sampling errors, which apply only to sample surveys; are due to the fact that only a subset of the population is selected. Non-sampling errors, which apply to all statistical processes is often categorised as coverage errors, non-response errors, processing errors and measurement errors. 6

7 Group work: what goes on? 7

8 On strengths and weaknesses of different data collection modes 8

9 Choosing the method of data collection Costs («Cash is King!») Timeliness Logistics Coverage Sampling Question format Complexity of Questionnaire Mode effect on quality Based on de Leeuw 2008, Chapter 7 in International Handbook of Survey Methodology. 9

10 Mode When choosing mode(s), the researcher must consider how the mode can affect validity and reliability (measurement errors) First analytically, and seek through relevant literature But also in many cases need to check this Bias exploration Bias adjustment 10

11 Unimode, Multi-mode or Mixed-mode? Multi-mode survey system The same respondents are interviewed in different modes at different stages. E.g. Household Budget Survey 1. Introduction interview (Face to face) days of diary (Self-completion) 3. Ending interview (Telephone) Mixed-mode Different respondents are interviewed in different modes at the same stage. E.g. The National Election Survey 50 percent of the net sample interviewed by face to face 25 percent of the net sample interviewed by telephone 25 percent of the net sample interviewed by paper questionnaire as a non-response follow up 11

12 Example of unimode, forecasting the election result on election night. Random digit dialling. Telephone interviewing. Gives a simple data matrix: Refnr V1_g V2_a V3_r V4_VLE V5_PLE V6_PPUE V7VCE V8PCE Vn n V1_g: gender V2_a: age V3_r: residence V4_VLE: Voted or did not vote last election V5_PLE: Party voted for last/previous election V6_PPUE: Party preference in upcoming election V7VCE: Voted or did not vote in current election V8PCE: Party voted for in current election 12

13 Example of a Multimode Survey System, Norwegian European Social Survey. A bit more complex data matrix: Sample Frame variables from registers Main interview Face to face Supplementary interview Self completion (paper) Refnr V1_g V2_a V3_r V4_VLE V5_PLE V690_VLE_ri V691PLE_ri Vn n V1_g: gender V2_a: age V3_r: residence V4_VLE: Voted or did not vote last election V5_PLE: Party voted for last election V690_VLE_ri: Voted or did not vote last election asked in reinterview V691_PLE_ri: Party voted for last election asked in reinterview 13

14 A mixed-mode matrix. The Norwegian Election Survey 2013 Sample Frame variables from registers Y from register Y s from survey Paradata Refnr V1_g V2_a V3_r V4_VLE_R V5_VLE_S V6_PLE Vn Mode F2f Tel F2f F2f Tel F2f SCp F2f Tel SCw SCp. n V1_g: gender V2_a: age V3_r: residence V4_VLE: Voted or did not vote last election from register V5_VLE: Voted or did not vote last election from survey V6_PLE: Party voted for last election 14

15 Types of mixed-mode designs (de Leeuw 2008) Contact Phase Mode Change Rational for implementation Effect on Survey Quality Advance notification in different mode than data collection Correct sampling frame Raise response Enhance credibility/trust Reduce coverage and nonresponse error No threats to measurement if data collection single-mode Recruitment/Screening/ selection in different mode than data collection Reduce cost Enhance efficiency Update/expand contact information for main mode Timeliness If pure screening, no threats to measurement If screening plus first part data collection in other mode potential mode effect on measurement 15

16 Macro flow chart of data collection in Statistics Norway, Face to face interviews. Mail Mail, SMS, , Telephone, personal visits. etc Telephone, etc Face to face interview 16

17 Types of mixed-mode designs (de Leeuw 2008) Response Phase Mode Change Rational for implementation Effect on Survey Quality Different (sample) persons by different modes when surveying on sample at one time period with one questionnaire Reduce costs Improve coverage Improve response Reduction of coverage and nonresponse error Mode effects on measurement confounded with subgroups Different parts of questionnaire by different modes when surveying one sample at one time point Improve privacy of measurement Reduce social desirability Improved data quality, especially with very sensitive questions Same persons with different modes at multiple time points (panel) Reduce costs Measurement differences causing confounding of time and mode effects Different (whole) samples by different modes, often at different times with different questionnaires Comparative research Different research traditions Different coverage Different cost structures Coverage error Nonresponse error Measurement error Incomparability 17

18 Types of mixed-mode designs (de Leeuw 2008) Reminders in different modes Rational for implementation Effect on Survey Quality From mode in which all respondents are asked to complete questionnaire Raise response Reduce nonresponse error If pure reminder no threats to measurement If reminder plus part data collection in other mode risk of potential mode effects on measurement 18

19 Advantage and disadvantages of Mail Surveys 1. Mail surveys lack the flexibility and interviewer support of interview surveys, which limits the complicity of the questionnaire. This is partly mitigated because visual stimuli, such as pictures or graphics can be used 2. Mail surveys are less intrusive than interviews: respondents may answer at leisure in their own time and there is no interviewer present who may inhibit free answers to more sensitive topics. Based on de Leeuw

20 Advantage and disadvantages of Mail Surveys 3. Lists with addresses of the target population should be available, telephone numbers are not necessary 4. Mail surveys normally takes longer time than telephone surveys, but often faster than face to face. 5. Mail surveys are less costly than both face to face and telephone surveys, and often require a much smaller field staff. Based on de Leeuw

21 Advantage and disadvantages of face to face surveys 1. Face to face interviewing has the highest potential with respect to types of questions and questionnaire complexity. Requires both well-trained interviews and well-tested questionnaires, and a qualified field staff is needed to take care of the logistics. This is very costly and time consuming and only worth it in some situations; carefully consider if all that potential is really needed to answer the research objective. Based on de Leeuw

22 Advantage and disadvantages of face to face surveys 2. Face to face interviewing has the highest potential regarding coverage and sampling, Also very costly, especially if the country is large and sparsely populated. Cluster sampling may be needed. if the sample dispersion is very high, telephone interviewing is often used instead. 3. The greatest asset of the face to face interview the presence of an interviewer is also its greatest weakness. Their presence may influence the answers respondents give, especially when sensitive to the total survey error due to variance in interviewer skill. Based on de Leeuw

23 Advantage and disadvantages of telephone surveys 1. Telephone interviews have less potential with respect to types of questions than face to face interviews, because there is no visual communication (unless materials can be sent in advance). Interviewers are available to assist the respondent and complex questionnaire may be used. However, fewer questions can be asked. The old rule of thumb has been minutes Still based on de Leeuw

24 Advantage and disadvantages of telephone surveys 2. Due to households that have no telephone, unlisted numbers, and mobile phones, coverage may be suboptimal. If good lists are available, telephone interviewing is comparable to face to face, from a sampling point of view. If the sample dispersion is very high, telephone surveys are often the only mode feasible. 3. Quality control is high as interviewers can be closely monitored and immediate feedback is possible. 4. Many interviews can be completed in a relatively short time at a lesser cost, using a smaller number of interviewers than face to face. Still based on de Leeuw

25 Advantage and disadvantages of Internet Surveys 1. Internet access varies strongly between countries and within countries. Coverage and Sampling may be suboptimal. Great benefit if a list of addresses of the target population is present. Mail addresses can be used to provide a url and a password. 2. Complex questionnaires and visual stimuli can be applied, but long questionnaires should be avoided 3. Like mail surveys Internet surveys are less intrusive and more private Based on de Leeuw

26 Advantage and disadvantages of Internet Surveys 4. Large numbers of completed questionnaires can be collected in a very short time and at a low cost. 5. Internet surveys can easily reach international populations 6. Questionnaire implementation must consider different devices, screen sizes and browsers used. Based on de Leeuw

27 On smartphones (and apps) Respondents have gone ahead of both survey methodologists and survey companies/ survey platforms. Without asking our permission they are filling out surveys from whatever device they have at hand at that moment Callegaro

28 Per cent of people using smartphones 100% Source: Google consumer barometer % 80% 70% 60% 50% 40% 30% 20% 10% 0% 28

29 Dealing with Mobile Web Most of our knowledge on how to make good questionnaires is based on experiences with PCs and big screens. Now: Web surveys = mobile web surveys The way we are thinking about good questionnaire design is changing. 29

30 Dealing with Mobile Web Smartphones and tablets increasingly similar, but differ on Screen size Connectivity:Wi-Fi and/or cellular Location: everywhere/home/special purpouse Ownership: personal vs. Shared Usage: short & frequent vs. longer, sustained Based on Link

31 Dealing with Mobile Web A new toolbox for survey researchers Voice SMS Videochat Web access Specialised applications Pictures and video Audio recording Scanning Bluetooth enabled (How) are we using these tools? How can we assess their utility and quality? Based on Link

32 Dealing with Mobile Web Ownership, feature use and comfort level varies a lot across Demographics Types of device Operative systems Device size This is, however, dynamic and changing Based on Link

33 Dealing with Mobile Web How to (re)design the mobile questionnaire? No changes to online survey Blocking mobile respondents Optimising a version for mobile platforms «Mobile friendly»/«mobile first» approach Build an app Decision can impact Response burden Nonresponse Measurement error No decisive answers yet, but the picture is becoming clearer Based on Link

34 Dealing with Mobile Web Mobile friendly/first: Start with the mobile design Shorter questions Reduce text Consider reducing response options Vertical response options Font size and spacing to facilitate touch screen options Design for what people need to touch first, rather than text Based on Link

35 Dealing with Mobile Web Mobile friendly/first: Make text boxes as large as possible Don t use fancy functionality Limit or discard graphics, logos etc. Never use Flash Always test on multiple devices Based on Link

36 Dealing with Mobile Web General advise for mobile web surveys Collect paradata from the user agent strings if you can Match your approach with the level of skill in the population you aim to investigate Make your survey as short as possible Avoid scrolling especially horizontally Balance question length and clarity Based on Link