Practical issues in the use of surveys in management research. George Balabanis

Transcription

1 Practical issues in the use of surveys in management research George Balabanis

2 contents When to use survey research Survey errors How to reduce survey errors and web surveys

3 When survey research can be used development of measures exploration of a phenomenon hypothesis and model testing experimentation

4 Types of survey research

5 Survey errors Sampling error Non-coverage error Non-response error Measurement error

6 Reducing sampling errors (certain members of the population are deliberately excluded from the selected sample) Random probability sampling Sufficiently large sample size

7 Reducing non-coverage error (certain members of the population are not included in the sampling frame. Coverage error is a function of both proportion of target population that is not covered by the frame and the difference between those covered and those not covered) Complete and updated sampling frames * sampling frame is the materials or devices that delimit, identify and allow access to the elements of target population

8 Reducing non-response error (members of the sampled population do not respond to survey questions. Non-response error is a function of the response rate and difference between respondents and nonrespondents) Improving response rates Making sure that those respond are not different from those that haven t Limited comparisons between respondents and non-respondents (or general population) Extrapolation (i.e., comparing early with late respondents, the Armstrong and Overton approach) Armstrong, J. S and Overton, T.S Estimating Nonresponse Bias in Mail Surveys. Journal of Marketing Research Vol. 14 no 3, p

9 What is a good response rate 50% is considered adequate (Babbie, 1990; Dillman, 1978) others suggest that 60% is needed (Fowler, 1984) 60% is considered to be good (Babbie, 1990) In practice, in most studies authors justify response rates with reference to obscure article or other articles in the area with lower response rates *Babbie E. (1990) Survey research methods Belmont CA: Wadsworth *Fowler F. F. (1984) Survey research methods Beverly Hills: Sage *Dillman (1978) Mail and telephone surveys: the total design Method. New York: Wiley

10 What is the normal response rate Median response rate: 31% (interquartile range = 21-46%). (28% if exclude studies with negotiated samples) median target sample size of n = 500 *based on an analysis of 154 studies published in the Academy of Management Journal, Strategic Management Journal, and Journal of Management between 1992 to 2000 Cycyota, Cynthia S.; Harrison, David A (2002) Top Manager Responses To Organizational Surveys: When Questioning Executives, Are Networks The Answer? Academy of Management Proceedings, pa1-a7

11 Tomaskovic-Devey, Donald; Leiter, Jeffrey (1994) Organizational survey nonresponse. Administrative Science Quarterly, Vol. 39 Issue 3, p p

12 How can someone improve response rates?

13 Greer, T. V Chuchinprakarn, N. and Seshadri S. (2000) Likelihood of Participating in Mail Survey Research: Business Respondents Perspectives, Industrial Marketing Management, vol.28, pp

14 Roth, Philip L.; BeVier, Craig A.. (1998) Response rates in HRM/OB survey research: Norms and correlates, Journal of Management, Vol. 24 Issue 1, p97-118

15 Claycomb, C.; Porter S. S. and Martin C.L. (2000) Riding the wave: Response rate and the effects of time intervals between successive mail survey followup efforts, Journal of Business Research vol.48, pp

16 Most common method cited in management research Dillman s Total Design Method recipients are more likely to respond if they expect that the perceived benefits will out-weight the perceived cost of responding Reduction of perceived costs (i.e., making questionnaire appear easier and less time consuming to complete) Increasing perceived rewards (i.e., making the questionnaire interesting) Increasing trust (i.e., using official stationery/ sponsorship) that the promised rewards will be realized

17 Some of the TDM design recommendations: ordering questions to assure that interesting ones related to the topic described in the cover letter come first; use of graphical design and numerous question-writing principles to ease the task of reading and answering questions (e.g. consistency in the use of large [capital or dark] letters and contrasting small letters); printing the questionnaire in a booklet format with a topically neutral but interesting cover; use of photo reduction of regular sized type to make pages seem smaller and easier to complete; use of four carefully spaced mailings, including a postcard follow-up one week after the original mailing, a replacement questionnaire and cover letter informing the recipient the questionnaire has not yet been received four weeks after the original mailing, and a second replacement questionnaire and cover letter seven weeks after the first mailing, sent to non-respondents by certified mail;

18 individually printed, addressed, and signed letters; Addresses, printed onto envelopes rather than on address labels; use of smaller than usual business stationery to reduce costs and make the entire request appear smaller and easier to comply with: cover letter content that includes descriptions of the study's social usefulness and why the respondent is important (repealed in different ways in each of the subsequent mailings); explanation of identification numbers and how confidentiality is protected.. More on Dillman (1978) Mail and telephone surveys: the total design Method. New York: Wiley

19 Reducing measurement error (the discrepancy between underlying, unobserved variables and observed survey response) Measurement error is influenced by: Questionnaire design Respondents Survey Mode

20 Reducing Measurement Error: Questionnaire design Open versus Closed Questions closed-ended questions can only be used effectively if its answer choices are comprehensive, and this is difficult to assure. Open questions: articulate respondents bias, giving the most salient possible responses,

21

22

23

24 Reducing Measurement Error: Questionnaire design Labelling of Rating-Scale Points Overall,how satisfied are you with strategy X? Extremely Dissatisfied Extremely Satisfied OR Extremely Dissatisfied Extremely Satisfied OR Extremely Mildly Mildly Extremely Dissatisfied Dissatisfied Dissatisfied Neutral Satisfied Satisfied Satisfied OR Extremely Mildly Mildly Extremely Dissatisfied Dissatisfied Dissatisfied Neutral Satisfied Satisfied Satisfied OR Extremely Mildly Mildly Extremely Dissatisfied Dissatisfied Dissatisfied Neutral Satisfied Satisfied Satisfied [ ] [ ] [ ] [ ] [ ] [ ] [ ]

25 Reducing Measurement Error: Questionnaire design RATINGS OR RANKINGS Please rank the following supermarkets in terms of the service they offer (1 being the best service, 5 the worst service) Tesco Sainsbury s Safeway Asda Morrison How would you rate the service of the following supermarkets Tesco very low very high Sainsbury s very low very high Safeway very low very high Asda very low very high Morrison very low very high

26 RESPONSE ORDER EFFECTS Please rank the following supermarkets in terms of the service they offer (1 being the best service, 5 the worst service) Tesco Sainsbury s Safeway Asda, Morrisons Which of the following factors is the influence your choice of a supermarket? Quality of products [ ] Selection of products [ ] Service [ ] Atmosphere [ ] Value for money [ ]

27 Other problems SOCIAL DESIRABILITY BIAS Anonymity Indirect questioning Use countervailing arguments ACQUIESCENCE Include negatively phrased questions

28 Reducing Measurement Error: Questionnaire pre-testing 1. Debriefing method small representative sample (10 to 25) discuss respondents experiences in a debriefing session Checking the questions that many respondents declined to answer 2. Cognitive pre-testing asking respondents to think aloud while answering questions, verbalizing whatever comes to mind as they formulate responses 3. behaviour coding an observer monitors pretest interviews (either live or taped) and notes events that occur during interactions between interviewers and respondents that constitute deviations from the script (e.g. the interviewer misreads the questionnaire, or the respondent asks for more information or provides an unclear or incomplete initial response).

29 Reducing Measurement Error: How respondents answer the questions Optimising 1. comprehension, 2. retrieval, 3. judgment, and 4. response selection. Satisficing Execute the above steps less thoroughly skip some of the steps Use heuristics Go for neutral points in rating scales or Don t know alternatives Randomly selects alternatives

30 Conditions that favour satisficing Satisficing is more likely to occur (a) the greater the task difficulty, (b) the lower the respondent s ability, and (c) the lower the respondent s motivation to optimize.

31 Reducing problems from satisficing Include screening questions Example, To what extent do you agree or disagree with following? I tried to answer the questionnaire honestly and accurately I had a hard time understanding this questionnaire This questionnaire is straightforward Response on a Likert scale: Strongly disagree, disagree, neutral, agree, strongly agree

32 Organisational research - Respondents Who should receive the questionnaire? Single Key informants (people who have the authority, capacity [knowledge] and motivation) Selection problems (i.e., just choosing the general manager or establish global competence of the respondent by asking tenure with the firm or self-assessment of knowledge, or establish task -specific competence with issues examined competence test) Problems Informant bias (differences in informants roles) Random error (memory failure, inaccurate recalls, hindsight bias, impression management)

33 Multiple informants discrepant responses due to different perceptions and knowledge Combining discrepant responses 1. Latent trait approach (reflective variables)* 2. Aggregation approach (use the average or weighted average (based on self-assessed confidence and competence)) 3. Consensual approach (ask respondents to resolve differences through discussion) 4. Hybrid approach *Kumar, Ajith, and William R. Dillon 1990 "On the use of confirmatory measurement models in the analysis of multipleinformant reports." Journal of Marketing Research, 27:

34 the latent trait method Bagozzi, R.P., Yi, Y., (1991) Assessing Construct Validity In Organizational Research, Administrative Science Quarterly, Vol. 36, Issue 3, pp

35 and web surveys

36 and web surveys Sampling and coverage errors Difficult to construct a sampling frame of or web users and select a random sample from it The proportion of the population (coverage rate) that can be reached by and web in many cases is still low (30% to 42% of the population) Internet population is different from general population: demographically (younger, richer and more educated) and possibly attitudinally Overcoming coverage problems Limit the study to those with internet access Making access available to those included in the sample

37 and Web surveys Non-response Errors Difficult to estimate response rates due to lack of sample frame (except in the case of surveys) Sampling from a list or through RDD recruitment to identify and enlist respondents (allow measurement of response rates; facilitates strategies targeted at the reduction of non-response and provide auxiliary variables for improvement of post-survey non-response adjustment) Legislation against unsolicited s and no random generation of addresses lead to prevalence of Internet panels recruited though other media (phone, mail), banner ads or open solicitations Response rates are lower in surveys than those of mail surveys Due to technical problems Due to lack of anonymity (especially when sensitive issues are researched)

38 /web surveys Increasing response rate of surveys Multiple contacts increase response rates Average response rates with a single contact: 28% Two contacts: 41% Three or more: 57% Personalisation (not using list servers, mailing lists, distribution lists, etc) Mixed method (include a postal address, telephone )

39 and web surveys Measurement Error Data quality same as mail surveys (although some reported more missing responses on web surveys) Difficulties in setting up a format that appears the same in all screens Lack of anonymity may generate social desirability bias Better response to open-ended questions Speedier response (4 days compared to 7 weeks) The appropriateness of a particular electronic questionnaire design should match objective of the study and audience Panel conditioning (time in sample bias)

40 Web surveys Sampling and coverage errors