Reasons for Unit Non-Response

Transcription

1 Reasons for Unit Non-Response Failure of the data collector to locate/identify the sample unit; Failure to make contact with the sample unit; Refusal of the sample unit to participate; Inability of the sample unit to participate (e.g. ill health, absence, etc); Inability of the data collector and sample unit to communicate (e.g. language barriers); Accidental loss of the data/ questionnaire. Unit non-response is defined relative to the eligible sample. In other words, if the sampling frame contains ineligible units, these do not contribute towards response/ non-response. Unit non-response is often divided into three components: non-contact, inability to respond, lack of co-operation (refusal). Response rates can usefully be divided into these (or other) components.

2 Item non-response A sample unit participates but data for some survey items are not available for analysis. Reasons could include: Refusal to provide an answer Inability to provide an answer Other failure to answer (e.g. by accident) Provided answer being of inadequate quality (e.g. incomplete, implausible, failing an edit/consistency check, etc.) Item non-response can be caused by: the action of the sample member (e.g. refusal to answer); the action of an interviewer (e.g. failure to ask a question that should have been asked, or failure to record the answer adequately); the survey design (e.g. poor routeing instruction). In practice, these factors interact.

3 Response Pattern We can define the survey response pattern by a matrix R = [r jk ], where r jk = 1 if item j is observed for unit k, and r jk = 0 otherwise. Possible response patterns include: Unit (k) y 1k y 2k y 3k y qk full response item non-response item non-response unit non-response Note that, due to item non-response, the set of units available for analysis depends on the item or items required.

4 Non-Response Error The effect of non-response on a survey estimate y r can be defined: y r = y n + nr n ( yr ynr ) where y r = statistic for the r responding units, y n = statistic for all n sample units, y nr = statistic for the nr non-responding units. (Groves, 1989, p.133). nr n ( y r ynr ) is the estimate (the realisation in one implementation of the survey) of total non-response error.

5 Components of Non-Response Error The non-response error has two elements: the non-response rate; the difference between responding and non-responding units in terms of y. Both of these elements are important. Note that the second element will result from both non-response bias and variance. In practice, studies suggest that bias is usually the main component.

6 Components of Non-Response Error ctd. Non-response errors can be divided into two components: Errors due to unit non-response Errors due to item non-response Each of these can in turn be decomposed into subsources, for example: Unit non-response due to: Non-contact Refusal to respond Inability to respond Item non-response due to: Routing (instrument) error Routing (interviewer) error Refusal to respond Inability to respond Etc.

7 Non-Response in Different Types of Surveys Survey structure: Cross-sectional Longitudinal Rotating Panel Flow sample Respondent type: Business Individual Household Individual within household

8 Cross Sectional Surveys Components of non-response: Refusals Non-contact Inability to respond Constraints on non-response: Limited field time Limited budget Survey population Survey task (burden)

9 Longitudinal Surveys Components of NR same as for cross-sectional surveys. But structure of NR different: Complete NR Wave NR (gaps) Attrition NR patterns depend partly on survey policy and partly on field efforts (especially "tracing").

10 Patterns of response to a 4-wave panel survey Policy 1: Issue all eligible cases at every wave. 16 possible patterns: Wave

11 Policy 2: Issue only wave 1 responding cases at each subsequent wave. 9 possible patterns: Wave Policy 3: At each wave, issue only cases responding to previous wave. 5 possible patterns: Wave

12 Example: England and Wales Youth Cohort Study Panel with 3 waves; Implements policy 1 (issue all eligible cases at every wave). 8 possible response patterns: Wave: No. of cases % of cases 1 8, , , , (From Lynn, Purdon, Hedges and McAleese, 1994)

13 Panel Attrition Specific to longitudinal studies. Can result from non-traceable or refusal cases Impacts can be quite different from cross sectional surveys Differences should be studied (using data from earlier waves). Design stage: strategy for dealing with attrition (both non-response and population exits ) Response rates as products of wave response rates

14 Panel Attrition ctd. Non-response due to mobility (e.g. sample unit has moved and cannot be traced): not unique to longitudinal surveys but can be especially problematic particularly if long time gap between waves Experience of the first wave may alter the response behaviour on subsequent waves: Almost unique to longitudinal studies Interview experience therefore very important

15 Survey Types and Data Collection Modes Interviewer surveys Face-to-face surveys Telephone surveys Self completion Mail surveys Web surveys Diary keeping Mixed mode strategies Supervised self-completion Mail questionnaire with interviewer follow-up Interview and diary keeping period Interview and self-completion Different modes for different waves in panel surveys

16 Survey Types and Data Collection Modes ctd. Face-to-face surveys typically yield higher response rates than telephone surveys. Response for mail surveys usually much lower than other modes of data collection. Differences are mainly due to level of respondent motivation possible in different modes. In mixed mode strategies: motivation of respondents vs. time and costs. Interview surveys: role of the interviewer is crucial. Telephone surveys: scope for interviewer influence is more limited. Mail surveys: only communication is via document design and content (q re, advance or covering letter, reminders).

17 Measuring and Presenting Response Rates Response rates measure the proportion of eligible sample units that successfully provided data, i.e. the reduction due to non-response of cases available for analysis. Response rates are more likely to be reported than almost any other survey process quality indicator. Response rates are often mistakenly used as a measure of quality of survey statistics. Remember, response rate is just one component of non-response error.

18 Refusals Factors influencing household survey participation (Groves & Couper, 1998): 1. Societal-level factors 2. Attributes of the survey design 3. Characteristics of the sample person 4. Attributes of interviewer 5. Respondent-interviewer interaction Social environment Survey design Household Interviewer Household-interviewer interaction Decision to co-operate or refuse

19 1. Societal-level factors The degree of social responsibility felt by sample persons Legitimacy of societal institutions The degree of social cohesion (Goyder, 1987). Interviewer respondent interaction: Particular persuasion strategies (on the part of the interviewer) Decision making strategies (on the part of the respondent) (Groves, Cialdini and Couper, 1992). 2. Attributes of the survey design The mode of the initial contact affects The number of channels of communication between interviewer and respondent (Groves, 1978). The selection of persuasion strategies to employ and the effectiveness of alternative strategies (Groves, Cialdini and Couper, 1992).

20 Other attributes: Perceived burden (e.g. length of the interview being requested, sensitivity of questions) Respondents level of interest and knowledge in the survey (topic, description) Incentives 3. Characteristics of the sample person Socio-economic characters Psychological characteristics Past experiences (Goyder, 1987). These factors tend to produce a set of pre-dispositions that affect the decision. They also affect the initial approach of the interviewer to the sampling unit. (Groves, Cialdini and Couper, 1992).

21 4. Attributes of interviewer Observable socio-demographic characteristics of the interviewer may affect the script evoked in the respondent s mind at the first contact with the interviewer. (Groves, Cialdini and Couper, 1992). No research has yet found strong links between stable interviewerpersonality characteristics and success in gaining co-operation. Reasons for this might be interviewers being relatively homogeneous group, tailoring, social skills, and other adoptive behaviours (Groves and Couper, 1998). Morton-Williams (1993) argues that social skills can be taught and offers an outline of such training. Those with greater interviewing experience tend to achieve higher rates of co-operation than those with less experience. It is still unclear, whether this is a selection effect (less successful interviewers terminate their employment earlier) or a training effect (due to the benefits of coping over time with diverse situations in recruiting respondents) or both (Groves and Couper, 1998).

22 There is some evidence that interviewers who, prior to the survey, are confident about their ability to elicit co-operation tend to achieve higher co-operation rates (Groves and Couper, 1998). 5. Respondent-interviewer interaction Persuasion strategies employed by the interviewer are determined not only by the interviewer s ability, expectations, and so on but also by features of the survey design and by characteristics of the immediate environment and broader society. Similarly, the responses that the sample person makes to the request are affected by a variety of factors, both internal and external to the respondent, and both intrinsic and extrinsic to the survey request. (Groves, Cialdini and Couper 1992).

23 Non-Contact A conceptual model for contacting sample households (Groves & Couper 1998): Social environmental attributes Physical impediments Socio-demographic attributes Accesible at-home patterns X Likelihood of contact Number of calls X Timing of calls Interviewer attributes Contact likelihood is a function of three factors: 1) Physical impediments that prevent visiting interviewers from alerting the household to their presence 2) When household members are at home 3) When and how many times the interviewer visits the household.

24 Reasons for Non-Contacts Association between when household members are at home and when interviewer calls: Life styles that lead to reduced time at home (patterns of work, related patterns of shopping and entertainment) Rural/urban differences Local crime rate (perceptions) Ethnic/cultural differences One-person households households with no children <5 or adults >70 Hidden refusals: Fear of crime/strangers/salesmen Too busy Answer machine screening

25 Example: The Survey Participation Process in the British Crime Survey It can be useful to identify the logical stages of the survey participation process. This was done for BCS (next page). It can be seen that there are many stages at which either a non-contact or a refusal could occur (see diagram on next page, from Laiho and Lynn, 2000). This illustrates the heterogeneity of the survey non-response phenomenon. Survey design and implementation features should be used to address each of these possible non-response outcomes.

26 Survey participation process in BCS (Laiho & Lynn 2000) Advance letter to the sampled address Contact attempt Office refusal Ineligible Dwelling Unit (DU) Multiple DU - Insufficent address - Not traced Single DU - Not yet built - Vacant/ derelicted / demolished - Empty - Business/industrial only - Other Listing of all adult (16+) members of the DU 1) List in systematic order by flat number 2) random selection of the dwelling unit Information about number of persons 16+ refused No contact with responsible/ any adult in selected DU (after 5+ attempts) Random selection of the respondent Contact attempt - respondent Respondent contacted No contact with selected person after 5+ attempts Completed interview Refusal Other reason for non-response Personal refusal by respondent Broken appointment, no recontact Proxy refusal on behalf of selected respondent Ill at home Away/in hospital during survey period Senile/incapacitated Inadequate English Other

27 Studying Non-Response Designs for Determining Characteristics of Nonrespondents: Special studies of non-respondents Using information on the sampling frame Asking others about non-respondents or having interviewer provide information about them (example on next page) Comparison of respondent characteristics by call number Comparison of respondent characteristics to census or other external information Studying persons who drop out of a panel survey after an initial interview

28 ALL RESIDENTIAL ADDRESSES (CONTACTS AND NON-CONTACTS INCLUDING VACANTS) 17. Does the address have an entryphone? Yes 1 No Which of the following are visible at the sampled address? CODE ALL THAT APPLY Burglar alarm 1 Security gate over front door 2 Bars/grilles on any windows 3 Other security device(s) 4 Estate/block security lodge/guards 5 None of these 0 INTERVIEWER ASSESSMENTS: 19. Are the houses/flats in this immediate area in a good or bad physical state? Mainly good 1 Mainly fair 2 Mainly bad 3 Mainly very bad Is the sampled house/flat in a better or worse condition outside than the others in this area? Better 1 Worse 2 About the same 3 Does not apply Do you know or think that the occupants are probably white 1 black 2 Asian 3 Other: 4 Don t know 5 22a. SAMPLED DWELLING IS: Whole house: detached 1 Semi-detached 2 IF NO DWELLING mid-terrace 3 SELECTED, CODE end terrace 4 FOR ADDRESS Maisonette 5 Flat: purpose-built 6 converted 7 Rooms, bedsitter 8 Unable to code 9 IF FLAT ETC (5-7 AT a.) ANSWER b-e. OTHERS - END b. CODE TYPE OF FLAT ETC: Self-contained 1 Not self-contained 2 Don t know 8 c. BUILDING HAS: Fewer than 5 floors 1 5 floors or more 2 Unable to code 3 d. FLOOR LEVEL OF MAIN ACCOMMODATION: Basement/semi-basement 1 Ground floor/street level 2 First floor 3 2 nd /3 rd floor 4 4 th 9 th floor 5 10 th floor or higher 6 e. BUILDING HAS: Common entrance: lockable 1 Common entrance: not lockable 2 No common entrance 3

29 Example: Using Interviewer Observation Data Data were collected by interviewers, using the form on the previous page, on the 1996 British Crime Survey. The following table uses the answers to question 22a. Response Selected Responding rate sample sample % % House: detached 82.6% semi-detached 79.6% end terrace 79.2% mid-terrace 77.7% Maisonette 74.9% Flat: converted 72.3% purpose-built 70.3% Rooms/bedsit 75.6% Unable to code 51.2% Base 13,117 10,059 Source: Lynn P (1996) Weighting for non-response, in Survey and Statistical Computing 1996, Chesham: Association for Statistical Computing

30 Example: Using Sample Frame Data These data are from the Scottish School Leavers Survey, a postal survey for which the sampling frame is a list of pupils and their school exam results. Highest Response Selected Responding Qualification rate sample sample % % 5+ Higher grades 91.1% Higher grades 85.1% Higher grades 81.7% Standard grades % Standard grades % Standard grades % Standard grades 4-7 only 62.6% No qualifications 59.6% Base 4,542 3,469 Source: Lynn P (1996) Weighting for non-response, in Survey and Statistical Computing 1996, Chesham: Association for Statistical Computing

31 Example: Using Geographical Data The following data are from the Italian Multipurpose Survey carried out by ISTAT. The figures presented are regression coefficients from a model to predict propensity to refuse. It can be seen that refusals are most likely in metropolitan areas and least like in small (rural) municipalities. Parameter s.e. Metropolitan area (0.147) Met. area ring (0.168) Large municipality (0.114) Small municipality 0 Source: Baldazzi et al (2002) Interviewer s effect on refusal risk in the Italian Multipurpose Survey: a multilevel approach, paper presented at the annual conference of the Italian Statistical Society, May 2002.

32 Example: Using Number of Calls These data are from the 1987 British General Election Survey. This kind of analysis can be done for any survey, even if there is no useful information from the sampling frame or from interviewer observation (e.g. postal or telephone survey). Interviewed at 1 st call Interviewed at 2 nd call Interviewed at 3 rd call Interviewed after 4+ calls Nonrespondents Social Class % % % % % Non-manual Unknown Manual Unknown Self-employed Unknown Unclassifiable Unknown Base 726 1, ,354 1,637 Source: Lynn P (1996) Weighting for non-response, in Survey and Statistical Computing 1996, Chesham: Association for Statistical Computing

33 Example: Comparing Non-Contacts, Refusals and Easy-to-get Households Data are from 1996 Health Survey for England. Lynn et al (2002) analysed calls record data and classified respondents as easy or difficult to contact, and willing or reluctant. It can be seen, for example, that persons who were difficult to contact were much younger than average and much more likely to be employed; but reluctant persons (potential refusals) were slightly less likely than others to be employed. Estimate Difficult to contact (6+ calls) A Reluctant (temporary refusal) B Hard-toget A+B Easy-to-get households C All responding households A+B+C Male (%) Age (mean) Owner-occupier (%) Employed (ILO definition) (%) White (%) Source: Lynn P, Clarke P, Martin J and Sturgis P (2002) The effects of extended interviewer efforts on nonresponse bias, in Survey Nonresponse (ed.s R M Groves, D A Dillman, J L Eltinge and R J A Little), New York: Wiley.

34 Common Non-Response Patterns 1. Many (most) household surveys find response rates lower in urban areas. 2. Many self-completion surveys find response rates higher amongst those with more education. 3. Interview surveys often find higher contact rates amongst many-person households than amongst 1- or 2- person households. 4. Many household surveys find contact rates higher amongst persons aged 65+, but cooperation rates lower. 5. In most (but not all) European countries, response rates are usually higher for women than men. 6. On most business surveys, response rates are higher amongst larger businesses. But These relationships may not be true for all countries, cultures and surveys.