Report on the Workshop in Monrovia, Liberia: Designing Labor Surveys and Measuring the Informal Sector April 1-15, 2009

Transcription

1 General Data Dissemination System (GDDS) Project Phase 2 Socio-Demographic Statistics for Anglophone Africa Report on the Workshop in Monrovia, Liberia: Designing Labor Surveys and Measuring the Informal Sector April 1-15, 2009 Edwin L. Robison Mathematical Statistician U.S. Bureau of Labor Statistics 1

2 Main Issues Main Outcomes Workshop Presentations PowerPoint modules delivered that were appropriate for workshop participants with different levels of statistical expertise. Concentration on priority 1, development of LF survey. Smaller groups and consultation on questionnaire evaluation/testing and sampling. Labor Force Questionnaire After training in the methods, a Liberialed expert panel was conducted and the draft questionnaire will be revised. Stressed the need for concepts to adhere to international standards and for questionnaire stability so that lf changes over time can be measured. Encouraged module approach with a core labor force module that can be used in other surveys Decision made to develop a child LF module since many core LF questions are not appropriate for children. LF Questionnaire Evaluation/Testing A plan with 3 phases of testing was drawn up: 1) informal in-house testing, 2) formal testing by interviewers, and 3) a small pre-test. A memo is to be prepared by Liberia. Labor Force Survey Sampling Methods Agreement to use the usual 2-stage design: 1 st -stage sample of EAs followed by 2 nd -stage sample of 1-20 households each Minimum of 240 or more households per region. Sample allocation spreadsheet developed. Spreadsheet for PPES selection of EAs developed. 2

3 Table of Contents 1. Summary and Main Findings 4 2. Workshop Modules 6 3. Labor Force Questionnaire Design, Evaluation, and Testing 9 4. Sample Design for a Household Labor Force Survey Other Statistical/Technical Issues Recommendations on Upcoming Priorities 19 Annex A Agenda for April 2009 Liberia Workshop 20 Annex B Participants, April 2009 Liberia Workshop 22 Annex C Terms of Reference 23 Annex D Daily Log 30 Annex E Labor Market Information Systems (Notes) 35 Annex F Simple Weighted Estimates (Notes) 56 Annex G Nonresponse Adjustment (Notes) 59 Annex H Random Groups Variances (Notes) 64 Annex I Sample Design Spreadsheet 66 Annex J Spreadsheet, Enumeration Area Sampling Within County 69 Annex K Workshop Module Handouts 70 3

4 1. Summary and Main Findings Under the auspices of the World Bank s General Data Dissemination System (GDDS) Project, Phase 2, Socio-Demographic Statistics Project for Anglophone Africa, a workshop was presented in Liberia April 1-15, The principal contact persons in the host country were Kehleboe Gongloe of the Ministry of Labor. Greetings and debriefings were participated in by: Dr. T. Edward Liberty, Director-General of the Liberia Institute of Statistics and Geo-information Services (LISGIS); John Josiah, Ministry of Labor, Deputy Minister of Administration; and Diavana Koikoi, GDDS coordinator for Liberia. The mission was organized by the World Bank (Ronald Luttikhuizen, Annette Kinitz, and Yayea Gloria Keleay) and Brian Graf of the U.S. Bureau of Labor Statistics. This mission s main purpose was to present a workshop concentrating on questionnaire design and sample design for household labor force surveys. Workshop modules were developed to benefit participants with varied backgrounds in questionnaire design and statistics. In addition, consulting was to be provided on 1) improving a draft labor force questionnaire and 2) developing a sample design and allocation for a labor force survey. Workshop modules were presented daily, and several time slots were set aside for consulting. We were guided by the agenda in Annex A, Agenda for April 2009 Liberia Workshop. The workshop modules and the agenda were prepared drawing from needs stated in the Terms of Reference (see Annex B), documents about the mission, and the priorities of Liberia. Modules that were delivered and other work adhered to the topics in the agenda, but not necessarily the order given (see Annex D, Daily Log). Specific topics were stressed or added and others deemphasized based on the situation found in the country. Main Findings, Outcomes, and Products The labor force questionnaire rough draft needed revision before further review, and evaluation testing. After training in the method, an expert panel was led by Daouda Kromah. Notes were kept on a question-by-question critique of the labor force questionnaire and Liberia will revise the questionnaire. After training on questionnaire evaluation/testing, a smaller group of participants was gathered and a three-phase plan was adopted for the forthcoming revision of the labor force questionnaire. To meet international standards and to facilitate analysis of economic progress over time it is of prime importance to maintain a stable core labor force module. Modules on special labor force topics are properly placed after the core labor force module. Core labor force module questions should not be modified to accommodate other topics, and questions to accommodate other topics should not be placed before or inserted into the core labor force module. 4

5 The development of labor force modules on special topics (informal sector, informal employment, child labor, etc.) should draw heavily from the experience of other countries. A standard 2-stage sample is proposed for a Liberian labor force survey -- a firststage sample of Enumeration Areas (EAs) followed by a listing operation and a second-stage sample of households (HHs) in those EAs. A sample design excel spreadsheet was developed for a Liberian labor force survey. The spreadsheet is driven by parameters such as number of HHs to sample, estimates of the average number of adults per household, and design effects. A national sample is allocated in proportion to county size, and a minimum sample size is ensured for each of 6 regions. Estimates of standard errors are made for the nation and regions. Reasonable national sample sizes and regional minimum sample sizes were discussed. Final sample sizes will depend on funding realities. Variance estimation is a vulnerability. When applied to complex surveys, standard statistical analysis packages usually understate variances and confidence intervals, and will mislead data analysis. More accurate estimates of variances are needed, and the use of the random groups methodology was suggested. It works very well for the type of sample used in the labor force survey and is usable for both national and regional variance computations. 5

6 2. Workshop Modules Handouts of the workshop PowerPoint slides are included in Annex K. The modules were developed so that the contents would be appropriate for workshop participants with different levels of expertise in statistics and questionnaire design. A technical module was generally previewed, presented the next day, and summarized the following day. The file names (without extensions) are given here. LiberiaA-intro An introduction to me, the consultant., and the major topics: The overall Labor Market Information System context Questionnaire design for labor force surveys Statistical design (sampling and weighting) and total survey error LiberiaB-LMISNotes The Annex E notes on Labor Market Information Systems were distributed to participants prior to the workshop. The notes touched on all topics to be covered in the following modules and served to give context and direction to the entire workshop. Main topics that were presented and discussed: The need for labor market information Key indicators (ILO) Determining what data is to be produced The need to adhere to international concepts, and developing country-specific concepts Questionnaires, and the decades of work needed to develop labor force concepts and questions that properly measure those concepts Statistical methods (sampling, weighting, and analysis) as part of the LMIS Mode of data release and the increasing popularity of the internet Data integrity and confidentiality Organization and scheduling Planning for success LiberiaC-TotSurvErr Total survey error can be decomposed into nonsampling error and sampling error. Sample surveys are not administered to the entire population and sampling error refers to the error arising solely from the sampling process, as measured by variance (or related measures such as the standard error or the coefficient of variation). Nonsampling error arises from other sources, such as nonresponse and processing errors, and raise problems for censuses as well as sample surveys. Nonsampling errors are usually thought of as causing biases, although the errors can cancel each other out. Various types of nonsampling errors are covered in the module, including unit nonresponse, item nonresponse, processing errors, and specification errors (differences between concepts to be measures and what the questions actually measure). Some statistical methods to control bias are presented. 6

7 A key element of the workshop were the frequent reminders of the importance of bias. In particular, poor questions can lead to bias. Ill-defined or undefined concepts can even make it impossible to determine if questions are measuring anything useful. A good survey needs both a good questionnaire AND good statistical methodology. LiberiaD-HHQuestDes This module gives a broad overview of questionnaire design for household surveys. Observe the country s general context and subject-matter context Define survey concepts and draft questions Evaluate questions then redraft questions, and it necessary re-evaluate Pre-test questions, and also conduct post-survey evaluations The tools of questionnaire evaluation are covered including expert panel review, debriefings and focus groups of interviewers (or respondents), scripted and unscripted follow-ups, and vignettes. Many pointers are given on what to watch out for when developing a labor force questionnaire. LiberiaE-HHSampDes The module starts with a general overview of sample design, then covers: Survey requirements (data needed and how low the variance needs to be) Some ways of determining sample size needs, including an introduction of the design effect concept a quantification of how variances from a complex survey differ from simple random sample variances Sample frame needs and the area frames used for many household surveys Standard sampling techniques of simple random sampling (SRS), systematic sampling (SYS), probability proportional to estimated size sampling (PPES), stratified sampling, cluster sampling, 2-stage sampling, and sample rotation LiberiaF-HH2-Stage This is a detailed presentation of the 2-stage household sample design that is used for household surveys in almost all developing countries. Area frames provide good coverage and a first-stage sample of enumeration areas provides good coverage of the households in a country. Examples are given of the typical PPES sampling used since enumeration areas vary in size. It is common to select a second-stage sample of households per each selected enumeration area. An explanation is given of the probabilities used for sampling at each stage and the computation of the 2-stage probability. The base weight of a household is the inverse of its 2-stage probability of selection. LiberiaG-HHWgt An introduction is given for household survey weighting procedures: Base weights the inverse of the selection probabilities Weight adjustments for nonresponse Benchmarking to population controls 7

8 LiberiaH-EstabFrame A discussion is given of list frames used for establishment surveys and typical sources of administrative data used to form those lists. Data elements included on the lists, and their usefulness in survey sampling, are covered. LiberiaI-EstabSamp A brief overview is given of establishment survey sampling and weighting. Comparisons are made to household survey sampling and weighting. LiberiaJ-RandGroup The module presents the random groups method of variance estimation that can be used for a complex survey. The subject is introduced by indicating the shortcomings of closed textbook formulas and the need for different methods for a 2-stage household sample with nonresponse adjustment and perhaps with population controls. The module concludes with a discussion of design effects. 8

9 3. Labor Force Questionnaire Design, Evaluation, and Testing A draft labor force questionnaire was prepared by Liberia prior to the workshop. Several of the workshop modules were customized to facilitate critiquing the questionnaire and developing an evaluation/testing plan. Labor Force Questions and Concepts Some familiar reservations were raised regarding international concepts/standards: Why are elder persons allowed to be classified as unemployed when they normally should not be working at a job? Why is a single hour of work enough to classify a person as employed? Why are children in school classified as employed, especially since they work few hours? In a subsistence economy, everyone has to work so there is no real unemployment. Some answers to these questions certainly boil down to standards are standards, but it was also noted that the standards as they exist today were developed with measurement issues clearly in mind. The one hour of work requirement to be classified as employed, for example, is a very simple requirement and the associated labor force question is a very simple one that gets more accurate answers than a more complex one would. Also, useful data analysis of changes over time is often not strongly affected by even seemingly substantial changes in concepts; it is more important that the concepts themselves remain stable. As a consultant, international comparability was kept in mind, the most conservative stance was taken, and changing concepts from accepted international standards was discouraged. In addition to producing data based on internationally defined concepts, the mission discussed the development of country-specific measures as long as these measures are clearly defined and explained. Country-specific measures can be very useful from the perspective of policy-makers and other users to develop a clear picture of the labor market in a given country. There was a pervasive misconception that international concepts/standards are restrictive. It is better to view them as minimum standards that must be met, but a country can go beyond them. However, labor force questions designed to enable data collection and production of labor force estimates consistent with international standards cannot be materially changed (though additional questions can be asked). Core Labor Force Module A modular approach to survey design was presented in the workshop. It is strongly recommended that a core labor force module be developed, then not changed in any fundamental way. A core module enables the publication of labor force status and some occupation and income statistics, but not much else. Stand-alone Liberia labor force surveys are proposed, but it is highly likely, due to resource constraints, that a labor force module will be attached to other surveys. 9

10 To the extent possible, the labor force questions should be adapted from questions already vetted for use in labor force surveys of other countries. Few countries have the manpower or experience necessary to do the extensive cognitive research required before changing those question. Once established, the core labor force module should remain nearly unchanged. For a labor market information system it is very important to keep concepts and questions as stable as possible so that real changes in the economy can be measured. Changes will affect response behavior and that in turn will damage comparability over time. In general, seemingly small changes in wording, ordering, and even module placement can substantially impact answers obtained. (Some care is needed in placement of the core labor force module in relation to other modules in a questionnaire. For example, it would be inadvisable to precede the module with other topics relating to the labor force.) Other Modules Children in the labor force It is very common for children to perform labor market work. Initial plans were to ask all labor force questions of children, but it was agreed during the workshop that a child labor force module concentrating on work activities would be developed by Liberia. Some adult labor force questions can be problematic when asked of young children, and some concepts inappropriate (such as unemployment). No age cutoff was agreed up on for the age cutoff, but some countries do administer all labor force questions to persons aged 14 and 15. Informal sector There is strong interest in the informal sector of the economy and in informal employment The Liberian economy is principally rural. In contrast to industrialized nations, there is a heightened awareness of the extent to which informal economic activities affect the overall economic picture of the country. There is a great desire to separately measure the informal sector and informal employment, but little know-how on how to approach the measurement problem. Existing international definitions are vague, particularly with regard to the informal sector. There is no accepted questionnaire module for household data collection that distinguishes between formal and informal that could be adopted. South Africa and other countries are working on it. It is hoped that the international community will further develop concepts and methods that can be used by other countries. Liberia is cooperating with Brazil in conducting cognitive research on the topic. Module Placement In a labor force survey or any other survey, the core labor force module should precede all other labor force modules. Preceding it with other labor force modules will change respondent behavior and the answers given. 10

11 Module placement must be kept in mind when developing questions for the modules. A particular question may seem more natural to insert before or into the middle of the core labor force module. It is emphasized again not to do that. Even if the question placement seems clumsy or the required question wording seems odd, it needs to go after the core labor force module. Liberia s General Context and Subject-Matter Context Good labor force questionnaire design requires an analysis of a country s general context and the subject matter (labor market) context. As part of the workshop, there were discussions on these topics. Liberia s Survey History Not extensive, but a population census was just completed. In-Person Interviewing Little phone interviewing or mailings. Official Language of the Questionnaire English, and interviewers often need to translate on-the-fly to other languages. Ethnography and Fluency Terms may be interpreted differently by ethnic groups, and the official language may not be well understood by large ethnographic subpopulations. Are there problems with translating some of the terms? If the problems are pervasive enough, consider translating the questionnaire into other languages. Who is the data needed for? For labor force surveys this is usually all adult persons. The appropriateness of many questions aimed at adults is questionable when asked of children. Special modules for children are preferable. Who are the appropriate respondents? Self response may be preferable for most adults, but who in a household is likely to be the best proxy respondent? Who is best suited to respond for older children and for younger children? Urban/Rural Differences Are households defined and identified differently? The best respondent for a city household may be the householder/owner, but in rural areas a different criterion may be needed. Does a term like business have the same meaning in a village as it does in the city? Concepts What are the definitions for country-specific concepts? Each question aimed at a concept should have a specified purpose. Subsistence Farming Will responses agree with conceptual definitions of work, and hours worked. Barter Will responses properly reflect barter as pay or profit? Expert Panel Conducted Expert panels were introduced in the workshop as part of a questionnaire designer s toolkit. Throughout the first week many things that questionnaire designers look for were mentioned. These were informed by my study of the draft questionnaire, but a direct analysis was never given by me. A few of these were: Last week is not always interpreted the same way as last 7 days Mixing reference periods (last 7 days, last month) can be confusing. Consistency of question formats Keep on track and don t mix in other questions with core labor force questions. 11

12 Some idea of critiquing questionnaires was given to the participants by looking at a selection of labor force questionnaires from other countries. Participants discussed what they did and did not like about the general appearance/format, specific questions, and the flow of the questionnaires. Taking the workshop participants as the experts, an expert panel was conducted. To make sure this was a Liberian effort, the expert panel was led by Daouda Kromah. He led a question-by-question critique and was responsible for taking notes. The discussion was very lively and participants did take note of most shortcomings in the draft questionnaire. Some of the most important observations/decisions were: Consistently reference last 7 days Use consistent wording in I/O questions Use the flow of the South African questionnaire Time use questions add confusion Remove children and instead develop a module for children Liberia is to re-draft the labor force questionnaire. The re-draft will be evaluated/tested. Further assistance may possibly be needed in developing a core labor force module. Evaluation/Testing Plan for Liberia s Labor Force Questionnaire Workshop presentations were given on questionnaire evaluation and testing. A smaller group met and formulated a plan for testing the re-draft of the labor force questionnaire. The 3-phase plan, to be written up by Liberia, is summarized here. Informal Testing LISGIS and Ministry of Labor employees administer the questionnaire to coworkers and family. It is important that parsons not working on labor force issues be included. Meet, evaluate the experience, and possibly revise the questionnaire. Formal Testing Have about 5 experienced and properly credentialed interviewers conduct about 10 interviews each. To control costs, the interviewers will probably live near Monroeville, but urban and rural respondents need to be available. Interviewers will choose their own respondents, but need to be given general instructions on obtaining persons representing the diversity of Liberia. Debrief each interviewer and bring them all together for a focus group. Revise the questionnaire and convene an expert panel to critique the questionnaire. Pre-Test This is more comprehensive than the previous testing, but can still be a small test involving 5-10 interviewers and about 10 households each. A sample should be drawn for each interviewer from a single enumeration area, and to control costs this can be done from census listings. Part of the evaluation will determine if interviewers had any problems finding the sampled households. Debrief each interviewer and bring them all together for a focus group. Another facet of the testing will convert the data on the questionnaires into the form needed for the survey (i.e., test data entry operations, industry/occupation coding and other related processes). 12

13 4. Sample Design for a Household Labor Force Survey Workshop presentations were delivered on household sample design. There were smaller group discussions on Liberia s sample needs for a labor force survey, and a spreadsheet created for a sample allocation with national and regional criteria. A spreadsheet was also created for sampling enumeration areas within a county. Two-Stage Area Sampling is Recommended A type of 2-stage area design, one that is in common use worldwide, is recommended. In the first stage, a sample of enumeration areas is sampled probability proportional to estimated size (PPES) within geographic strata, where the EA size is the number of households obtained from the latest census. In the second-stage a fixed number of households is systematically sampled from each selected EA. It is recommended that 10 households be sampled within each selected Enumeration Area. Sampling between households per selected EA is very common worldwide. Geographic Stratification Although deciding on proper geographic stratification is necessary, it is not strictly a statistical issue. In the workshop, the choice was presented as being a data-driven decision. For what geographic units (strata) is basic labor force data desired? Liberia has 15 counties divided into 6 regions. Most surveys have subnational needs in addition to national data needs, and the merits of regions versus counties were discussed. It was decided to design a survey that provides basic labor force data for each region by assuring that the allocated sample size exceeds a specified minimum. The counties would be a secondary geographic stratifier. For example, each county would be directly allocated sample, but no provision made for a minimum sample size. Subregion stratification of rural areas, since basic rural labor force data is needed for each of 8 subregions. (The Kampala subregion is entirely urban). Note: For urban stratification to mesh well with rural stratification, it makes sense to control urban sampling and estimation by subregion or to actually stratify by subregion. In particular, valid estimation of subregion totals requires each subregion to have some urban sample. Guidelines on Minimum Samples in the 6 Regions Each geographic stratum needs enough sample to allow the publication of basic labor force data. Labor force data deals not only with large characteristics (employed), but with relatively rare characteristics (unemployed or underemployed). It is common to want not just labor force totals for a stratum, but some detail by gender, age, and other demographics. A good rule-of-thumb is to require minimum stratum samples of households (20-40 EAs if 10 households are sampled in each of those EAs). 13

14 Take a minimum sample allocation of 300 HHs per stratum as an example. Assuming an average of 2.5 adults per HH, that is a minimum of 750 adults in sample. Assuming a 90% response rate, about 675 adult responses can be collected from each stratum. For a characteristic comprising about 3% of the adult population, about 20 responses will be obtained. Even allowing for the complex sample, a 90% confidence interval around 3% will be approximately (2%, 4%). The proposed minimum sample size allows for some meaningful between-stratum comparisons of labor force estimates. The per-stratum minimum of 750 in-sample adults of the example is nearly the same as the 700 that I recommend for the United States Current Population Survey (CPS). We do not routinely publish sample estimates of labor force for strata with samples that small, but use the CPS estimates as inputs to time series models. Type of Sample Allocation -- PPES The recommended allocation of sample is as follows: PPES, sample allocated proportionally to county size But with a minimum of about 300 households per region In Probability Proportional to Estimated Size allocation, larger strata will be allocated larger samples. Stratum A will be allocated twice as many households as Stratum B if its estimated size is exactly twice as large Often PPES allocation is simply referred to as Probability Proportional to Size (PPS) allocation. In practice, the stratum sizes are usually estimates, for example the sizes of the strata at the time of the last Census and not their actual current sizes. The size measure will be number of households from the last census. Alternative size measures were discussed in the workshop. The use of the projections that more accurately reflect the current distribution is preferable to the direct use of Census figures. For labor force data there is a slight statistical preference for using the working-age population as the measure of size. The average number of adults per household may differ across strata, and allocations based on HH measures of size will be slightly less efficient. However, the average number of working-age persons per HH varies little across the country, so the choice between using HHs as the size measure and using working-age persons as the size measure is not critical. There are more important considerations than the loss of efficiency: availability of the size estimates, currency of the size estimates, and capability of making reasonable adjustments for stratum growth. (A single multiplicative adjustment for growth applied to the entire country does not improve sample allocation.) Since geographic strata were determined based on data needs, each stratum needs a minimum number of HHs in its sample. PPES allocation of n sampled HHs will often leave the smaller strata with too few sampled HHs. 14

15 Assuming that 10 HHs are sampled from each sampled EA, allocation of HHs is equivalent to allocation of EAs. The spreadsheet discussed later allocates EAs and multiplies by 10 to obtain HH allocations. Except by luck, allocations do not calculate to whole numbers; rounding to the nearest whole number is usually sufficient. Type of EA Sampling -- PPES Within each geographic stratum, it is recommended that Enumeration Areas be sampled probability proportional to estimated size. General comments about the size measure to use for PPES allocation also apply to sampling. It is not necessary for the estimated EA sizes to add up to the stratum sizes used for allocation. It is not even necessary for the same type of size measure to be used for EA sampling. However, it is good survey practice to use consistent size measures for allocation and sampling, and to ensure additivity of those measures. The natural ordering or sort of EAs within a geographic stratum will be used. This keeps nearby EAs together. Alternatives were discussed in the workshop, since countries often sort EAs using census economic variables correlated with labor force variables such as unemployment, average housing prices, and measures of poverty. Sample Design Spreadsheet The most important product produced during this mission was a sample design spreadsheet in Excel (Annex I), customized for examining options specified by Liberia. The spreadsheet was developed in collaboration with Ministry of Labor and LISGIS staff. Kehleboe Gongloe is thoroughly familiar with the spreadsheet. The sample design spreadsheet is driven by parameters: seed number of HHs for national allocation (5,000 in Annex I) number of HH per EA (10 in Annex I) adults per HH (3 in Annex I) response rate (.9 in Annex I) design effect (2 in Annex I) CI factor for confidence intervals (1.645 in Annex I for 90% confidence intervals) minimum HHs to sample per region (440 in Annex I) Given the parameters: The number of EAs for national allocation is computed (500 = 5,000/10 in Annex I) and that number of EAs is allocated to the counties. The minimum number of EAs per region is calculated (44 = 440/10 in Annex I), and that number of EAs is allocated to the counties in regions that need more than the national allocation. (So the total allocation in Annex I is EAs or 5,454 HHs.) For proportions that can be changed (.1 and.6 in Annex I), variance measures are calculated for the nation and regions based on the national allocation alone and for the national allocation supplemented with regional minimums. 15

16 The question most often asked during the workshop was: What size sample is needed for a labor force survey? A simple formula based on national criteria was given, but there is no easy formula to apply when both national and regional criteria come into play. A spreadsheet like the one developed makes it easy to modify parameters and examine different scenarios. The spreadsheet error measures include national and regional estimates of variances, standard errors, coefficients of variation, and confidence intervals for a large proportion (.6) that can be associated with employment and a small proportion (.1) that can be associated with unemployment or underemployment. unemployment, underemployment, and employment. These error measures make it very easy to determine if a specified sample size meets the primary needs of labor force analysis. The calculations express a variable as a proportion of the working-age population. Simple random sampling (SRS) variance formulas are used in each stratum, modified by the response rate and design effects. If a variable such as unemployment is 10% of the working-age population, then.1 is the proportion to use. The simplest SRS-type variance formulas to use for stratum i is N i 2. (.1)(1-.1)/n i, where N i is the working-age population and n i is the number of working-age persons in the sample (not the number of households). The formula de i N i 2. (.1)(1-.1)/r i modifies the SRS formula by using a design effect de i and replacing the sample n i with the number r i of responding working-age persons. A national variance estimate is obtained by summing stratum variance estimates. Design effects represent the increase of variance for a complex sample compared to a simple random sample. (Or, for national variances, compared to a stratified simple random sample.) When applied to standard errors, as in the spreadsheet that was developed, the square root of the design effect is taken. Coefficients of variation (the standard error of an estimate divided by the estimate itself) and confidence intervals on the spreadsheet also include the design effect. Spreadsheet for Sampling Enumeration Areas A spreadsheet was developed to assist in sampling Enumeration Areas within counties. The spreadsheet in Annex J is a short example with many fewer EAs than the spreadsheet developed during the workshop. For each EA, an estimated size is needed, and the Cumulated Size (CUM) column is calculated. The number of EAs to sample must be supplied, and based on that a sampling interval (SI) is calculated. A random start (rs) is also needed, and the spreadsheet calculates rs+si, rs+2si, rs+3si, etc. The first EA to select is the unit where CUM exceeds rs, the second EA to select is the one where CUM exceeds rs+si, the third EA to select is the one where CUM exceeds rs+2si, etc. 16

17 5. Other Statistical/Technical Issues Random Groups Variance Estimation Variance estimation is a vulnerability since standard analysis packages do not usually produce the correct variances for complex surveys. The packages typically understate variances and standard errors, and as a consequence confidence intervals are too narrow. Economic analysis can suffer, for example, when a data item is compared between demographic groups a false statistically significant differences may be identified when more precise variance information would not draw the false conclusion. Similarly, false statistically significant differences can be found when comparing the same data item from surveys administered in different years. The 2-stage samples for labor force surveys are on the mild side of complex, nevertheless they are complex and an appropriate method of variance estimation is needed if statistically valid comparisons are to be made within geographic domains and between domains. The workshop module and the notes in Annex H guided discussion on the topic. The text Introduction to Variance Estimation by Kirk Wolter is recommended. The use of the random groups methodology using R=10 groups was suggested. It is the simplest of the variance estimation methods that are appropriate for complex surveys. The method works very well for the type of sample proposed for a Liberia labor force survey. It is usable for national and regional estimates of variance, although the method will be much more stable at the national level. Not only must each random group be a subsample that in all important respects resembles the full sample, but each random group must be processed through the same weighting steps. For example, if full-sample estimates are benchmarked or controlled to population figures external to the survey; each random group must be separately benchmarked to the same population figures. Design Effects It is possible to compute design effects for important data items that quantify how different (usually how much larger) a complex sample s variances are than simple random sample variances for a sample of the same size. Design effects vary from one data item to another, and a valid method of estimating a complex survey s variances is needed to reliably compute design effects. Weighting The advantage of using simple weighted estimates in preference to formula-driven estimates was covered in the workshop and additional notes are provided in Annex F. A very good reference book for sample survey estimation and weighting is Sampling Techniques, 3 rd edition, by William G Cochran. The need for nonresponse adjustment was raised, and a simple weighting class method was introduced in the workshop. The weight adjustment is of the form #HUs- 17

18 eligible/#hus-responding. Partially responding HUs where most adults are assigned a labor force classification should be included in the denominator. For notes on the topic, see Annex G. Listing Housing Units in Enumeration Areas, and Eligibility The thoroughness required for household listing in selected Enumeration Areas was discussed. If a survey immediately follows a census, it is common to use the census listing. It is also common to designate HHs as eligible for the survey if they were occupied at the time of the census. Given a longer delay (from the census), the following changes are needed: HHs relisted in selected EAs Sampled HHs might be vacant/destroyed, and eligible HHs would be those currently occupied at the time of the labor force survey. An intense and expensive census-type relisting operation is planned for selected EAs. For example, a listing of every HH that includes visits to determine the householders and survey eligibility. Two other options were discussed. An even more intense listing for marginally extra cost that gathers some usable household data for statistical purposes. A less expensive listing that does not gather householder information or determine eligibility. This might work in some urban areas, since all that is needed for sampling is a list of HHs and a way to find the ones you want after sampling. But in rural areas it is difficult for interviewers to locate sampled HHs without information on the householder. 18

19 6. Recommendations on Upcoming Priorities Liberia has plans for producing Labor Market Information meeting international standards. This mission to assist the country was sponsored by the World Bank (General Data Dissemination System Project, Phase 2, Socio-Demographic Statistics Project for Anglophone Africa). Progress was made on developing a labor force questionnaire and designing a sample allocation for a labor force survey. However, the bulk of the time was devoted to delivering a workshop, so opportunities for hands-on consulting were limited. Areas needing further work are noted here. Based on the expert panel that was conducted, the draft labor force questionnaire will be revised. Also during the workshop, a questionnaire evaluation and testing plan was adopted. It will be very helpful if the revised questionnaire is reviewed by labor force survey experts before commencing evaluation/testing. The sample design proposed for a labor force survey is a 1 st -stage sample of enumeration areas followed by a 2 nd -stage sample of households. A parameter-driven spreadsheet was developed for allocating sample to counties. Several options are possible, and these can be discussed with me via . Variance estimation was identified as a vulnerability. Software packages, including Stata that Uganda uses, usually underestimate actual variances of surveys with complex sample designs. Random Groups variance estimation was covered in the workshop, but there was insufficient time to work on applying the method. There is strong interest in developing questionnaire modules on the informal sector of the economy and on the participation of children in the labor force. 19

20 Annex A Annex A Agenda for April 2009 Liberia Workshop Workshop on: Designing Labor Surveys and Measuring the Informal Sector Labor Statistics Module GDDS Phase II Project World Bank Monrovia, Liberia April 1-15, 2009 Wednesday, April 1 9:30 am Introductions and Administrative Issues Edwin Robison, Supervisory Mathematical Statistician Statistical Methods Staff Office of Employment and Unemployment Statistics US Bureau of Labor Statistics 10:30 am Overview of Developing Labor Force Surveys 1:00 pm Lunch 2:30 5:30 pm Group discussion Progress made to date on developing LFS Plans for LFS Needs Problems Thursday, April 2 9:30 am Survey Design 1:00 pm Lunch 2:30 5:30 pm Work session and Discussion Survey Design Friday, April 3 9:30 am Data Tabulation 1:00 pm Lunch 2:30 5:30 pm Work sessions and Discussion Data Tabulation Monday, April 6 9:30 am Sample Design 1:00 pm Lunch 2:30 5:30 pm Work sessions and Discussion Sample Design Tuesday, April 7 9:30 am Sampling and Variance Estimation 1:00 pm Lunch 2:30 5:30 pm Work sessions and Discussion Sampling and Variance Estimation 20

21 Annex A Wednesday, April 8 9:30 am Questionnaire Content and Design 1:00 pm Lunch 2:30 5:30 pm Work sessions and Discussion Questionnaire Design Thursday, April 9 9:30 am Questionnaires: Uganda and South Africa 1:00 pm Lunch 2:30 5:30 pm Work sessions and discussion Questionnaires Friday, April 10 9:30 am Data Collection 1:00 pm Lunch 2:30 5:30 pm Work sessions and Discussion Data Collection Monday, April 13 9:30 am Data Processing 11:00 am Work sessions and Discussion Data Processing 1:00 pm Lunch 2:30 pm Data Analysis 3:30 pm Work session and Discussion Data Analysis Tuesday, April 14 9:30 am Dissemination of Data 11:30 am Work session and Discussion Data Dissemination 1:00 pm Lunch 2:30 5:00 pm Discussion: Experience of Uganda Wednesday, April 15 9:30 am Designing and Implementing a Pilot Survey 11:00 am Wrap-up/Planning for Success 1:00 pm Lunch 21

22 Annex B Annex B Participants, April 2009 Liberia Workshop Workshop on: Designing Labor Surveys and Measuring the Informal Sector Labor Statistics Module GDDS Phase II Project World Bank Monrovia, Liberia April 1-15, 2009 Ministry of Labor Thomas M. Diggs Kehleboe Gongloe, workshop coordinator Patrick T. Kenyor Daouda Kromah Urias C. Paye George H. Saah Erica D. Senkpeni Kwie T. Yorke LISGIS Diavana Koikoi, GDDS coordinator for Liberia Yusuff M. Sarnoh Bebee Smith Wesley Francis F. Wreh IPS/UL Musu Pusah Ywalla Statistics Sierra Leone Nabbie Bangura 22

23 Annex C Annex C Terms of Reference General Data Dissemination System, (GDDS phase 2) Socio-Demographic Statistics Project for Anglophone Africa: Provision of technical assistance as the expert for: Topic: Labor Statistics Liberia, Ministry of Labor (MOL) Background With financial support from the Department for International Development (DFID) of the United Kingdom, the World Bank is implementing a project to assist 21 Anglophone Africa countries to participate in the General Data Dissemination System (GDDS). Participating countries are being assisted to participate in the GDDS through two separate, but linked projects both financed by DFID. The IMF is providing project management and technical support in the area of economic and financial statistics. The World Bank is providing technical support in the area of socio-demographic statistics. Both projects run concurrently until June Technical Assistance Technical assistance is being provided through the World Bank to help countries implement plans for improvements in population, health, agriculture, labor market, justice and security, management of statistical systems, GIS and small area statistics. The GDDS framework developed by the IMF provides the framework for the detailed elaboration of long-term statistical development strategies. Participating countries have already expressed their requests for technical assistance and both the IMF and the World Bank have developed their assistance strategies. Liberia was one of the countries which asked for technical assistance in the field of Labor Statistics. Terms of Reference Background Liberia attended the GDDS2 Module launch workshop on Labor Statistics in Kampala in May 2007, where it drew up a Country Work Plan regarding three technical assistance missions covering three priorities. These priorities are part of the Work Plan Structure Doc. The purpose of the Work Plan Structure Doc is to act as a living document for the duration of the technical assistance and to serve as an information base from which the TOR for each mission can be drawn up. To this end, this TOR for 23

24 Annex C the first mission to Liberia has been drawn up from the work plan and discussions with Kehleboe Gongloe, MOL, to define technical assistance needs. The Ministry of Labor s general objectives regarding the topic are to design and develop a sustainable labor force survey, to review the existing establishment survey conducted in Monrovia, and to incorporate questions to measure informal sector and informal activities into the LFS. Purpose of the assignment The purpose of the assignment is to complete the first technical assistance mission to provide support with designing and developing a labor force survey for Liberia. The country has identified the following priorities (table): 1. Develop labor force survey (LFS) 2. Review establishment survey 3. Incorporate questions into LFS to measure informal sector activities See annexes for the details on the topic and the timing. This mission will cover of the pre-set priorities in varying degrees o Priority 1: 100% o Priority 2: 0% o Priority 3: Will be included with Priority 1 The total consultant time for the mission is 15 days divided as follow: o o o 11 days actual mission time 3 days consultant preparation time 1 days report writing The first draft of the report will be due 2 weeks after the completion of the mission. Following are the objectives and planned activities by priority for the mission: Priority 1: Development of Labor Force Survey Objectives o Develop a household labor force survey to generate reliable data on the labor force. Activities o Review and discuss work completed to date on labor force survey; o Review and discuss plans for labor force survey; 24

25 Annex C o o o o o o o Assess household sampling frame and evaluate suitability of frame to be used for selecting a sample of households; Discuss sampling procedures; Assist authorities in selecting and finalizing a sample of households (if necessary); Review and discuss questionnaire, and assist as needed; Develop plans for pilot testing of questionnaire; Conduct training program for MOL staff and staff from Sierra Leone; and Develop detailed work plan to guide authorities. Priority 2: Review establishment survey Objectives o Expand coverage of existing establishment survey to include areas outside of Monrovia. Activities o All activities during this first mission will focus on development of the LFS and measuring informal sector. Priority 3: Measurement of informal sector Objectives o Collect and disseminate data on informal sector and informal activities. Activities o LFS will be designed to include questions on informal sector (units) and informal activities. Skill requirements The consultant needs relevant labor statistics experience and skills within the African context and needs to read and write English fluently. Deliverables: The deliverables are listed by priority: Deliverables for Priorities 1 and 3: o o o o o Assessment of work completed to date with recommendations; Assessment of sampling frame with recommendations; Assessment of questionnaire with recommendations; Conduct training program on development of labor force survey; and Work plan to guide authorities. 25

26 Annex C Deliverables for Priority 2: o No deliverables for Priority 2 100% of mission to focus on Priorities 1 and 3 and the development of a LFS questionnaire (including questions to measure informal sector and informal activities. A concluding Mission Report as well as Mission Evaluation forms will form part of the final deliverable. Communications: The consultant will meet at the start of the visit with the head of the organization and GDDS coordinator, if possible, and will report briefly to them at the end of the visit. Duration As noted, the total consultant time for the mission is 15 days with 11 days mission time, 3 days preparation time, and 1 day report writing. Timing To be completed during the period March 27 April 16, Annexes: Country Work Plans 26

27 Annex C Country: Liberia Priority 1 Topic: Developing a Household Survey 1. Problem to be addressed: Design of the household survey 2. Strategic objective(s): To know how to run a household survey 3. Activities required: Sampling design, concepts and definitions, questionnaire design, data collection. Data processing, analysis and dissemination. 4. Input required (international): Technical assistance on design of household survey 5. Own preparations required: Providing basic documents. Research on design of surveys. 6. Output planned: To be able to conduct a household survey and establish relevant labour market information. 7. Changes anticipated: Labour information from household 8. Linking with further activities: Labour market module in all household surveys 9. Other donors supporting this topic: ILO 10. Timing: 50% from October

28 Annex C Country: Liberia Priority 2 Topic: Modify Existing Sample Survey 1. Problem to be addressed: Need to expand sample survey from Monrovia to include other parts of the country 2. Strategic objective(s): To know the number of existing establishments, wages, hours, worked and salaries 3. Activities required: Sampling design, concepts and definition of sample, modify existing sample survey 4. Input required (international): Technical assistance on design of existing survey, review of existing sample survey 5. Own preparations required: Recommendations on area of importance, making available documentation on existing survey, getting the sample firm for the whole country 6. Output planned: Planned indicators and other indicators for the whole country; wages and salaries 7. Changes anticipated: Representative sample survey for the whole country 8. Linking with further activities: Labour market module information 9. Other donors supporting this topic: ILO 10. Timing: 30% from January next year