Sampling. Building Evidence on Results- Based Financing for Health 3 rd Annual Impact EvaluaAon Training Workshop Bangkok, October 17-21, 2011

Transcription

1 Building Evidence on Results- Based Financing for Health 3 rd Annual Impact EvaluaAon Training Workshop Bangkok, October 17-21, 2011 Data Quality Assurance: Sampling, Ques6onnaire, Fieldwork Beatriz Godoy and Juan Muñoz Sistemas Integrales October 18, 2011 Sampling Health faciliaes and households should be selected through a documented process that gives each unit in the populaaon of interest a probability of being chosen that is posi6ve and known This permits making inferences from the sample to the enare populaaon with known margins of error (external validity) Samples are generally not simple random samples Samples are instead StraAfied by Region, by Urban/Rural, by IntervenAon / Control, Selected in two stages or more Area Units in the first stage/s Households in the last stage 2 1

2 Sampling error Sampling error is the result of observing a sample of n households (the sample size) rather than all N households in the populaaon of interest The standard error e is a measure of a sample s precision The chances for the true value of an indicator being farther than 2e apart from its sampling esamate are about 95 percent The standard error e decreases with the square root of the sample size n. To reduce the error to one half, the sample size must be quadrupled The size of the populaaon N has almost no influence on the size of the sample that is needed to achieve a given precision To obtain naaonal esamates, big countries and small countries require samples of about the same size Increasing the sample size will generally reduce sampling errors However, it is also likely to increase non- sampling errors 3 Sampling techniques Why two stages? IntervenAons are generally focused on faciliaes An updated list of all households is generally unavailable A single- stage sample would be too sca\ered in the territory Why straaficaaon? In order to potenaally improve precision, by gaining control of the composiaon of the sample In order to provide esamates for subgroups that would otherwise be poorly represented (small regions, women- headed households, etc.) Two-stage sampling solves these problems, but the sample becomes less precise as a result of clustering These two objectives are generally contradictory in practice Most stratified samples select households with unequal probabilities. This implies that the survey needs to be analyzed with weights. The combined result of clustering and stratification is called design effect 4 2

3 Sampling - Design effect (deff) Our survey will typically have a complex design, with two stages, stratification, etc. deff = n Our Survey n A Simple Random Sample with the same precision Deff depends on the indicator being measured For socio-economic indicators it is typically 3 or more It can be a little less for demographic indicators It can be a lot more for infrastructure indicators Deff depends on the cluster size Socioeconomic surveys try not to exceed households per cluster Demographic surveys may occasionally do more Sample frames An adequate sample frame needs to be available before a sample can be selected A sample frame is a list of all units in the populaaon The sample frame for the first stage is generally the most recent list of faciliaes (or census enumeraaon areas) The sample frame for the last stage is generally developed specifically for each survey, by way of a household lisang operaaon conducted in all sample points. The Ame and budget of household lisang are Small enough to be considered a marginal part of the overall data collecaon effort Large enough to be a headache if they are forgo\en or underesamated 3

4 Sampling - Household lisang issues Household lisangs (and the subsequent selecaon of the households to be visited) can be prepared by the same fieldworkers who will conduct the interviews, or by independent enumerators The choice is difficult. Sample points larger than a few hundred households may require segmentaaon The sample point is divided into smaller areas of approximately equal size called segments. Then one (or maybe a few) of the segments are randomly selected and listed. SegmentaAon is a de facto extra sampling stage that is very difficult to supervise. It should only be used as a last resort. Beware of imitaaons and shortcuts Implicit lisang (a.k.a. random walk ) consists of asking the interviewers to select every n- th household on the ground rather than on paper. It is not a recommended opaon. Sampling - Nonresponse None of the following is a soluaon for nonresponse Replace nonrespondents with similar households Increase the sample size to compensate for it Use correcaon formulas Use imputaaon techniques (hot- deck, cold- deck, warm- deck, etc.) to simulate the answers of nonrespondents The best way to deal with nonresponse is to prevent it Some of the above may have a prevenave value A\riAon Things that happen out there in the world Things that happen to us in the project 4

5 Asking the right quesaons Are the relevant topics covered? Is each topic adequately covered? Discussions with key stakeholders Indicators needed For monitoring (MDGS, Etc.) For impact evaluaaon Review of previous surveys Review of relevant qualitaave studies Reality checks PiloAng Field- tesang To avoid respondent faague: Maximum 90 minutes per respondent in a single sikng 9 Choosing the respondents Each quesaon should be addressed to the person who is be\er informed to answer Different respondents may need to be interviewed in each facility or household Who signs the le\er of consent? Proxy respondents to be avoided 10 5

6 Assuring comprehension Wording Avoid ambiguous wording Ask one quesaon at a Ame (don t ask double- barreled quesaons) Be clear who the quesaon refers to Make the reference period explicit Make measurement units explicit Avoid jargon or academic phrases Straighmorward language (avoid double negaaves) QuesAons should be culturally sensiave and appropriate TranslaAon Interviewers to read quesaons as they are wri\en Back- translaaon is needed DiagnosAc tools (Peabody, Mc Arthur, etc.) should be adapted to local context 11 The flow of quesaons QuesAons should be asked in a logical order QuesAons should only be asked when applicable The decision of when and to whom a quesaon should be asked should never be len to the interviewer judgment This is done by establishing an explicit skip pa\ern 12 6

7 Draning issues QuesAon types Closed form Yes / No MulAple answer Numeric Textual Don t knows (Don t provide perverse incenaves for them) QuesAon numbering Graphic organizaaon Free form Grid Flap Sonware tools Excel much be\er than Word Use formulas for quesaon numbering AutomaAc translaaon 13 Reality checks - PiloAng Designers and analysts should be engaged Sample does not need to be large or random (emphasize quality over quanaty) PiloAng is an iteraave process 14 7

8 Fieldwork and data management Why is good fieldwork important? Basically, because bad fieldwork can be disastrous Survey Month Households reporting illnesses, accidents (Q407) Households reporting chronic deseases (Q401) Households reporting agricultural activities (Q901) Number of crops reported (Q911) Households Households Total Nb of Nb of HH reporting reporting fishing durables having credit livestock activities (Q924) (Section 12) (Section 13) activities (Q918) HH size mean Number of lines with food consumption 1 1, , ,538 1, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Fieldwork and data management Fieldwork organizaaon Organized into teams Supervisor 2-4 Interviewers Data entry operator Anthropometrics / Bio- tesang professional A realisac survey plan Who visits which locality and when Few teams working for a longer period, rather than many teams working for a short period A realisac work plan Specialized teams (for facilities and for households) or dualrole teams? What to do in each locality and how long does it take to do it Time is needed to find the correct respondents Time is needed for quality assurance Teams need Ame to move between localiaes 16 8

9 Fieldwork and data management Fieldworker selecaon Need to consider Gender Age Language EducaAon (sufficient, but not much more) Compliance with naaonal health regulaaons Willingness to accept the realiaes of fieldwork Etc. SelecAon requires CV scanning Face to face interviews Be\er train more than needed, and select the best aner training 17 Fieldwork and data management Fieldworker training A well- defined training plan (measured in weeks, not in days) Senior speakers for plenary sessions A group of master trainers to conduct the group sessions in parallel Lodging, catering and transportaaon Equipment and supplies Households for field pracace Space for plenary and group sessions 9

10 Fieldwork and data management Quality assurance Human supervision Visual scruany of completed quesaonnaires Observing interviewers during fieldwork Check- up interviews Computer- based checks (Computer Assisted Field Edits: The CAFE approach) Use the computer to detect inconsistencies while the team is in the cluster and can correct them by re- visiang the households Timely available datasets are also useful for Monitoring fieldwork Analysis of paraal datasets Opportune decision- making Avoid data cleaning Need to implement adequate data transmission and concentration protocols Reliance on external fieldwork implementaaon firms makes the above paracularly challenging The bo\om line Gekng good quality data is Important Difficult Time- consuming Needs to be accounted for at the Ame of Planning BudgeAng 10

11 21 11