Don't Build That Survey Yet: Ask These 10 Questions First Summit 2017 Carol Haney

Don't Build That Survey Yet: Ask These 10 Questions First Summit 2017 Carol Haney

KNOWING THE END GAME

1 The Turn data into information end is a very good place to start Create your tables, graphs, charts etc. before you write your survey - what is your end deliverable?

1 Think Turn data into information of others Share your placeholder end deliverables with your stakeholders

2 Analysis Everything should have a purpose Plan Know who and what you are measuring

2 Analysis Everything should have a purpose Target Segments Plan 63% 17% 20% Females, 18-34 Males, 18-34 Other Know the characteristics of the subgroups of respondents that are of interest to you

2 Analysis Everything should have a purpose Plan Image of Good-Enough Know what good-enough means to you and your stakeholders

Who qualifies?

3 Customer Know your population (coverage) Image of person purchasing something off of a shelf Define your customer base Purchased one or more products or services in the past 5 years

3 Customer Know your population (coverage) Map of United States How many customers meet this criteria? 10,122 50,122 100,122 500,122 1,000,122

3 Potential Know your population (coverage) Image of person purchasing something off of a shelf, perhaps reaching for a different brand Customer Define what a potential customer looks like Lives in the United States, is 18 years or older and has shopped one or more times at a competitor in the past year

3 Potential Know your population (coverage) Customer IR Incidence rate supports access to a large enough sample to statistically support results Lives in the United States = 325M (2016) 18 years or older = 209M (2014) 64% Shopped at a competitor in the past year = 31M (Market Share Data, 2016) 15%

4 Sample Sampling your population 500,122 Customers 50,122 Customers in the sample Size Number of individuals to interview does not need to be an absolute proportion of the population

4 Sampling your population Sample Size Confidence: 95% Error: 2.2% Number of individuals need only be large enough so that the respondent subgroups are large enough to show statistical significance and aggregated in as a total sample

4 Sampling your population Sample Size stratified sample Random sampling allows for the right number of individuals invited to survey allows for the sample size listed

4 Sampling your population Quotas Image of people, standing in lines of different lengths Quotas allow for limiting the number of respondents to fill the subgroups of interest

4 Sampling your respondents Quotas Confidence: 95% Error: 2.2% Quota sampling allows for the right number of individuals invited to survey allows for the sample size listed Note that bias is introduced when not using technique of random sampling, yet may be sufficient given possible means

RESPONSE RATE VS. COMPLETION RATE Response rate: Total number of completes Total number of invites vs. Completion rate: Total number of completes Total number of completes + total number of starts

5 Know your population (non-response) Who isn t responding? Image of a scale, with one side much more weighted than the other Source of bias in the final sample, those who do not respond may have characteristics that are of interest to you and impactful on your analysis

Survey Content

6 Standard Questionnaire Design Structure Use standard structure of questionnaire that allows for respondents ease of answering (preamble, screener, general, specific, demographics)

6 Questionnaire Design Question/Response Image of burglar Reuse (steal) questions already pretested and found methodologically sound, where ever possible

6 Questionnaire Design? Image of person who looks confused New Questions For new questions, review for confusion, mistakes, potential cognitive biases, questions that are not answer questions that are part of you end deliverable

7 Respondent Interpretation Precision Image of a person choosing between two doors Don t leave it up to the respondent to try to interpret what you mean: use precise definitions and err on the side of too much precision

Data Management

8 Restructured Data Coding: Restructuring data Image of a Qualtrics with an additional column being added Understand the structure of your variables and robustly restructure specific variables to perform analysis if necessary

8 Data Coding: Weak Variables Only data that matters Image of a Qualtrics with a column being hidden Know and be wary of weak variables, that is, those variables that are operational, untested or have no variation to the distribution, etc., and possibly remove from the dataset

9 Data Data Editing: Quality Responses Analysis Image of a race car Check for quality responses in completed data, such as analyzing data for straight-liners, poor openends, gibberish, speeders

9 Maximize Data Editing: Quality Responses responses Image of a person who is slightly blurry and then also is completely in focus Look at partials and see if they can be added to the complete dataset without compromising quality

10 Is Adjustment: Weighting weighting needed? Image of scale Weight the data if the final, collected and cleaned data does not match the known and verified distribution of specific characteristics of the population of interest

10 Adjustment: Weighting How to weight Vary the value of each response in the dataset to match specific characteristics of the population of interest; use rim weighting if your weights require multi-variate adjustment

10 Don t Adjustment before Analysis game the system Do not analyze the data in order to determine what should be adjusted or manipulated ( hacking )

11 What s Baker s Dozen Analysis: Star-Gazing significant? Do not rely solely on significance tests to determine what your data have to say about your research question; instead, determine whether the effect or difference you see is significant for your research

11 Baker s Dozen Analysis: Fishing What s significant? Do not test many different comparisons or models in search of significant results; instead, only analyze and report on those that are part of your analysis plan

Total Survey Error

NO ONE IS PERFECT Who qualifies: Errors of non-observation Survey Content: Errors of observation TOTAL SURVEY ERROR Data Management: Errors of processing

NO ONE IS PERFECT Who qualifies? Survey Content Data Management ALL ARE EQUALLY IMPORTANT

NO ONE IS PERFECT Who qualifies? Survey Content DO YOUR BEST Data Management

PRESENTER + Firstname Carol Lastname Haney PRODUCT MANAGER // QUALTRICS carolh@qualtrics.com

HOW WAS THIS SESSION? Text @S64 to 35134