Monica H. Swahn, Ph.D., MPH Associate Vice President for Research Professor, Public Health Partnership for Urban Health Research Georgia State

Transcription

1 Monica H. Swahn, Ph.D., MPH Associate Vice President for Research Professor, Public Health Partnership for Urban Health Research Georgia State University Associate Director for Research, Emory Center for Injury Control

2 What is survey research? Pragmatic considerations Types of surveys Measurement Data collection methods Sampling Coding Interviewing kills Data entry

3 Any type of research involving eliciting a response from a participant or participants. Best method to collect data for describing a population too large to observe directly. Surveys include the use of a questionnaire an instrument specifically designed to elicit information that will be useful for analysis. Mail questionnaires Telephone interviews Internet surveys Face-to face interviews

4 What are the goals of the project (research question)? Manuscript preparation Clinical guidelines Inform training practices/curriculum Policy development What is the population of interest? What data is really needed (analysis plans)? What resources are available to support the survey? How will the data be used? How will it be collected? How will it be analyzed? Does it need to be linked to other data sources (e.g., medical records?) Will data collection occur once or multiple times?

5 1. Define construct Define key constructs and what should be measured A construct is a concept or idea used in educational theories. Typically complex abstractions (e.g., motivation, self-esteem, creativity ) 2. Select or develop measure (instrument): operationalize construct (how should it be measured?) An operational definition is the specific procedure used to produce and measure a construct.

6 Find existing measures Previous research (articles) Collections of measures Ongoing data collection Government recommendations Determine quality of measures Validity Reliability Developing new measures

7

8

9

10 Measures should be valid and reliable. Validity refers to truthfulness; a valid measure is one that measures what it claims to measure. Example: Do exams in your courses test the material covered in course lectures and texts? Reliability refers to the consistency of a measure. For example, a measure is considered reliable when different observers consistently agree about an observation or when the same measure yields similar results each time it is administered Note that a measure may be reliable but not valid. e.g., a scale that consistently underreports someone s weight is reliable but not valid.

11 Face validity Does the measure look like it gets at the construct, on the face of it? Does it seem to make sense? Content validity Does the measure include all the important dimensions of the construct? Does it capture the full range of the construct? Predictive validity Does the measure predict logically related outcomes or behaviors in the future? Convergent validity Does the measure correlate with other, closely related measures in the same data set? Discriminant validity Is the measure independent of (not correlated with) other measures in the same data set that it does not logically relate to?

12 In research, the term reliability means "repeatability" or "consistency". A measure is considered reliable if it would give us the same result over and over again. If our measure, X, is reliable, we should find that if we measure or observe it twice on the same persons that the scores are pretty much the same.

13 Test-Retest Reliability Administer the same survey to a large sample of people at two different times. The correlation between the two tests should be strong (r.80) if the measure is reliable (consistent). Split-Half Reliability Taking half of the items on a test and assessing the correlation (Spearman- Brown adjustment) between the two half-tests; prefer a r.80 Internal Consistency: Cronbach s alpha (α) Statistically compute the correlation between each item on a test (inter-item correlation). Items should be correlated with each other signifying the test is consistently measuring the construct of interest

14

15 Dichotomous Questions: When a question has two possible responses, we consider it dichotomous. Surveys often use dichotomous questions that ask for a Yes/No, True/False or Agree/Disagree response. There are a variety of ways to lay these questions out on a questionnaire:

16 Nominal: we might measure occupation using a nominal question. Here, the number next to each response has no meaning except as a placeholder for that response. Ordinal: refer to preferences or other ordering where ordering itself matters.

17 We can also construct survey questions that attempt to measure on an interval level. One of the most common of these types is the traditional 1-to-5 rating (or 1-to-7, or 1-to-9, etc.). This is sometimes referred to as a Likert response scale.

18 Another interval question uses an approach called the semantic differential. Here, an object is assessed by the respondent on a set of bipolar adjective pairs (using 5-point rating scale):

19 Typically used for exploratory research and are considered qualitative data. Participants responds in their own words. Examples: Describe a typical day?; Tell me about your relationship with your boss? What do you think about the election process? Answers need to be recorded and coded for themes. Special software are available to provide guidance like Nvivo. It is a very different approach than the quantitative data analyses used by standard survey questions.

20 Do you need all participants/patients/people in a population? (Census) If not, you will need to select a sample from the larger population Should it be representative of the population? Does it need certain characteristics? How many people will you need? (goals of project?) How to select a representative sample? Random selection, generated by a sampling frame and a random numbers table Sampling frame: list of every unit (individual, school, teacher, etc) in the population. Random numbers table assures that each member of the population has an equal and independent chance of being selected.

21 Simple random sample - a sample in which every member of the population has an equal chance of being chosen the best choice when you have no other reasonable basis for selecting individuals to study may be difficult if population is very large does not guarantee that systematic differences in the population will be represented in the sample Stratified random sample - a sample in which specific subgroups are represented in proportion to their appearance in the population the population is divided into subpopulations (strata) representatives of each subgroup are then chosen randomly good when you want more detail about specific groups

22 Cluster Sampling Design Used when intact groups (e.g., schools, classrooms, districts) cannot be separated into small units (e.g., individual students) Can be randomly selected in the same manner as individuals Multistage Sampling Design Random selection at 2 or more stages (e.g., randomly selecting 200 school districts in the U.S., then randomly selecting 20 schools in each district, then randomly selecting 10 classes within each of these 20 schools = three-stage random sampling design)

23 Being truly random in your selection of participants can be harder than it sounds. Non-probability sampling - a sampling technique that is not random, but is still systematic and (usually) theoretically motivated no way to estimate margin of error not as fair as probability sampling Convenience sample - a sample composed of individuals from a population that are readily available and willing to respond generally involves self-selection to participate restrictions on characteristics of participants may vary (but are usually few)

24 Data collection strategies need to be considered in conjunction with study population and data collection method as well as costs: Can participants answer the questions themselves? Do they have access to the internet or a phone? Each strategy has strengths/weaknesses and implications for costs! Self-administered surveys Mail Internet Interviewer-administered surveys Face-to-face Telephone

25 Standardized Survey Interviewing Minimizing Interviewer-Related Error and Bias General Guidelines Familiarity with Questionnaire Training, training, training! Following Question Wording Exactly Recording Responses Exactly Appearance, demeanor, building rapport

26 In every research project involving data collection, generate a codebook that describes the data and instructions for use and access. Minimally the codebook should include the following items for each variable: variable name variable description variable format (numerical, text) instrument/method of collection date collected respondent or group notes

27 What the participant sees: How often during the last year have you had a feeling of guilt or remorse after drinking? What the data entry person sees: How often during the last year have you had a feeling of guilt or remorse after drinking? Never Less than monthly Monthly Weekly Daily or almost daily 0=Never 1=Less than monthly 2=Monthly 3=Weekly 4=Daily or almost daily

28 Method of data collection impacts how you will proceed with data entry and analyses. Example, for internet or app based surveys, data will already be entered. If other method, need to make plan for data entry and database system.

29 Babbie, E. (2004). The Practice of Social Research 10 th Ed. CA: Wadsworth. Cottrell, R., & McKenzie, J. (2005). Health promotion & education research methods: using the 5 chapter thesis/dissertation model. Boston: Jones & Bartlett. Trochim, W. (2006). The Knowledge Base. Remler, D.K. & Van Ryzin, G.G. (2010). Research Methods in Practice: Strategies for Description and Causation. CA: Sage Publications.