CBMS Methodology LOUIS-MARIE ASSELIN

Transcription

1 IMG Institut de Mathématique Gauss 9 Carré F.X. Lemieux Lévis, Québec, Canada G6W 1H img@globetrotter.net CBMS Methodology Module 5 CBMS as a Monitoring Tool for PRSP LOUIS-MARIE ASSELIN CBMS MIMAP International Workshop Hanoi, January This document has been produced with the financial support of the MIMAP programme, IDRC, Canada.

2 Louis-Marie Asselin CBMS Data Collection and Processing Techniques Louis-Marie Asselin IMG and CECI Module 5 CBMS as a monitoring tool for PRSP Introduction Discussion paper: Poverty Reduction Source Book Monitoring and Evaluation, sections 1, 3.1, 4, 5. World Bank G. Prennushi,G. Rubio and K. Subbarao, 2001 Module 5 CBMS as a monitoring tool for PRSP Main topics > categories of indicators for PRS(1.2) > disaggregation dimensions (1.3) > data requirements (1.5) > specific issues (1.7) Module 5 CBMS as a monitoring tool for PRSP CBMS: a very large data set for a national poverty profile Aggregation issues: two types. > vertical aggregation: to build a national poverty profile from community level data > horizontal aggregation: to build a composite indicator from multiple indicators Both require data reduction techniques Vertical aggregation techniques: optimal sampling > The CBMS provide an extremely rich and updated sampling frame for a national survey on poverty > The output of such a survey, for PRS monitoring: a national multidimensional poverty profile > The sample design effect: need to be controlled for optimal precision Optimal sampling: sample design effect > Discussion paper: Does sample design matter for poverty rate comparisons? S. Howes and J. Olson Lanjouw Review of Income and Wealth, 44, #1, 1998 > A summary of basic theory Basic Sampling Techniques Louis-Marie Asselin, CECI and IMG, 1999 CBMS Data Collection and Processing Techniques 1

3 Louis-Marie Asselin Optimal sampling: sample design effect > Using the CBMS as a tool for numerical estimation and analysis: A Simulation Laboratory for the sampling effect on Poverty Profile estimation > A case study: the Vietnam MIMAP Simulation Laboratory Horizontal aggregation techniques: building a composite indicator > Discussion paper: An operational tool for evaluating poverty outreach of development policies and projects M. Zeller, M. Sharma, C. Henry and C. Lapenu IFPRI, FCND Discussion Paper # 111, June 2001 Horizontal aggregation techniques: building a composite indicator > A technical proposal: Composite Indicator of Multidimensional Poverty Louis-Marie Asselin, CECI and IMG, June 2002 CBMS Data Collection and Processing Techniques 2

4 Draft for Comments. April, 2001 Monitoring and Evaluation Outline G. Prennushi, G. Rubio, and K. Subbarao 1. Setting Up a Poverty Monitoring System 1.1 Defining goals, indicators and targets 1.2 Selecting indicators 1.3 Disaggregating indicators 1.4 Setting targets 1.5 Determining data requirements 1.6 Determining the frequency of monitoring 1.7 Elements of poverty monitoring systems that often need attention 2. Designing Impact Evaluations 2.1 Deciding when to conduct an impact evaluation 2.2 Measuring the impacts of policies and programs 2.3 Determining data requirements 2.4 Obtaining data 3. Challenges Ahead for Monitoring and Evaluation 3.1 Assessing the process of formulation and implementation of poverty reduction strategies 3.2 Evaluating the overall poverty impact of poverty reduction strategies 4. Strengthening M&E capacity and feedback mechanisms 4.1 Strengthening capacity 4.2 Strengthening feedback mechanisms 5. Promoting Participation in Monitoring and Evaluation Guide to Resources

5 Draft for Comments. April, 2001 Technical Notes TN 1. Major types of evaluations TN 2. Impact evaluation design TN 3. Impact evaluation methods for policies and full-coverage programs TN 4. Types of data sources for impact evaluation Case Studies CS 1. Monitoring the Progress of the Poverty Eradication Action Plan in Uganda CS 2. Monitoring the Poverty Reduction Strategy in Tanzania CS 3. Citizen Feedback Surveys as a Tool for Civil Society Participation in Assessing Public Sector Performance: The Case of Bangalore in India CS 4. Evaluating the gains to the poor from workfare: Argentina s Trabajar program CS 5. Evaluating Bolivia s social investment fund CS 6. Impact of active labor programs: Czech Republic CS 7. Impact of credit with education on mothers and their young children s nutrition: Lower Pra rural bank program in Ghana CS 8. Evaluating Kenya s agricultural extension project CS 9. Evaluating Nicaragua s school reform: A combined quantitative-qualitative approach CS 10. The impact of alternative cost-recovery schemes on access and equity in Niger CS 11. Schooling outcomes in Philippine elementary schools: Evaluation of the impact of four experiments CS 12. Assessing the poverty impact of rural roads projects in Vietnam

6 Draft for Comments. April, 2001 Summary The objective of this chapter is to assist countries in developing a system to monitor and evaluate whether a poverty reduction strategy is effective in reducing poverty. How do we know if a poverty reduction strategy is effective? First, a poverty monitoring system is needed to track key indicators over time and space and see if they change as a result of the strategy. Thus, Section 1 of the chapter discusses how to set up a poverty monitoring system: how to define key indicators, track them over time, and see what changes have taken place. Many countries already have poverty monitoring systems in place, so the task is to assess their adequacy and strengthen them as necessary. Experience shows that elements such as the tracking of public expenditures and outputs and quick monitoring of household well-being need special attention. Also, participatory data collection methods and qualitative information give a different perspective and should not be overlooked. Second, rigorous evaluations should be done selectively to assess the impact on poverty of interventions that are key components of the strategy. Section 2 of the chapter looks at how to decide when it makes sense to do a rigorous impact evaluation, and how to design and carry out an impact evaluation, including what data are needed for different methodologies and how to get the data. Other types of evaluation, such as assessing the process of formulating a poverty reduction strategy, can also be useful. Section 3 discusses this topic briefly, as there is only limited experience so far. The section also briefly discusses another challenging topic: evaluating the impact of poverty reduction strategies as a whole, as opposed to the impact of specific components of a strategy such as programs or single policies. The key point made here is that a solid monitoring system will provide the basic data necessary to conduct such evaluations, should the need arise in the future. Both monitoring and evaluation activities need to be carried out by institutions that are competent and that have strong links to key decision-makers, if they are to be useful in the design and implementation of a poverty reduction strategy. Much monitoring and evaluation takes place without adequate development of in-country capacity and without strong links to key decisionmaking processes; thus, precious opportunities to learn what works and what does not are lost and funds are sometimes. Section 4 offers guidance on how to build capacity and in particular strengthen the processes that provide policymakers and others with feedback on the impact of policies and programs. A key message of this section is that dissemination of results is critical for use. Results that are not widely disseminated, through mechanisms tailored to different groups in civil society, will not be used, and the resources that were spent in getting such results will be wasted. Non-governmental actors be they research institutions, civil society organizations, specialinterest and advocacy groups, or others have an important role to play in the design of the monitoring and evaluation system, in actually carrying out monitoring and evaluation activities, and in using the results. Section 5 discusses the role of these actors. A Guide to Resources at the end of the chapter contains references to other sources of information. Technical Notes and Case Studies provide more detail on specific topics and country examples. 3

7 Draft for Comments. April, Setting Up a Poverty Monitoring System 1.1 Defining goals, indicators and targets Before a monitoring system can be set up to assess whether a poverty reduction strategy is effective in reducing poverty, it is necessary to agree on what poverty reduction goals the strategy wants to achieve, select key indicators and set targets for such indicators. Goals, indicators, targets There are probably many possible definitions of these terms, but the following are used in this Sourcebook: Goals are the objectives a country or a society wants to achieve; they are often expressed in non-technical, qualitative terms, such as eradicate hunger or reduce poverty. Indicators are the variables used to measure progress toward the goals. For example, progress toward eradicating hunger could be measured by looking at the number of families who say they are not able to have three meals a day all twelve months of the year. Targets are the quantified levels of the indicators that a country or society wants to achieve at a given point in time for example, all families should be able to eat three meals a day all twelve months of the year by Example: the International Development Goals The International Development Goals (IDGs) provide an example of the types of goals, indicators, and targets that can be used to monitor progress. Following various international conferences of the mid-1990s, the Organization for Economic Cooperation and Development (OECD), the United Nations, and the World Bank, in partnership with developing countries, established explicit goals and targets to monitor progress in poverty reduction (see box 1). Box 1. International Development Goals, Indicators and Targets For economic well-being: Reduce by half the proportion of people living in extreme poverty by The goal is to reduce poverty; the indicator selected is the share of people living on less than $1 per day, and the target is to reduce this share by half by the year Other indicators were selected to track progress in poverty reduction the poverty gap, the poorest fifth's share of national consumption, and the share of children under five being underweight but targets were not set for these indicators. For social development: Achieve universal primary education in all countries by The indicator selected is the gross enrollment ratio, and the target is 100 percent by Eliminate gender disparities in primary and secondary education by The indicator selected is... Reduce by two-thirds the mortality rates for infants and children under five and by three-quarters the mortality rates for mothers by (Indicators and targets are clear here) Provide access to reproductive health services for all individuals of appropriate age no later than The indicators selected is contraceptive prevalence. 4

8 Draft for Comments. April, 2001 For environmental sustainability and regeneration: Implement a national strategy for sustainable development in every country by Reverse trends in the loss of environmental resources by It is important to note that the IDGs were chosen to monitor progress at the global level and guide development assistance; they are not meant to determine which goals individual countries should choose. 1.2 Selecting indicators Once a set of goals have been agreed upon through participatory processes, the next step is to identify indicators also in a participatory way to measure progress toward those goals. 1 As shown in Figure 1, indicators can be broadly classified into two categories: intermediate and final. When an indicator measures the effect of an intervention on individuals well-being, we call it a "final" indicator. For example, literacy may be considered one of the dimensions of wellbeing, so an indicator measuring it say, the proportion of people of a certain age who can read a simple text and write their name would be a final indicator. Sometimes final indicators are divided into outcome and impact indicators. Impact indicators measure key dimensions of well-being such as freedom from hunger, literacy, good health, empowerment, and security. Outcome indicators capture access to, use of and satisfaction with public services, such as use of health clinics and satisfaction with the services received; access to credit; representation in political institutions and so on. These are not dimensions of well-being in themselves, but are closely related. When an indicator measures a factor that determines an outcome or contributes to the process of achieving an outcome, we call it an input or output indicator, depending on the stage of the process in other words, an "intermediate" indicator. For example, many things may be needed to raise literacy levels: more schools and teachers, better textbooks, and so on. A measure of public expenditures on classrooms and teachers would be an input indicator, while measures of classrooms built and teachers trained would be output indicators. What is important is that inputs and outputs are not goals in themselves; rather, they help to achieve the chosen goals. Outputs differ from outcomes because they are fully under the control of the agency that provides them; so for example the number of schools built is an output, because it is directly under the control of education or other public authorities, while the number of children going to the schools is an outcome, because it depends on the behavior of children and their families. Table 1 gives examples of intermediate and final indicators related to key goals. 1 This chapter takes the goals as given. See the Chapter on Participation for a discussion of participatory goal-setting. 5

9 Draft for Comments. April, 2001 Figure 1: Types of indicators GOAL: Achieve universal primary education Final indicators IMPACT OUTCOMES Effects on dimensions of well-being Literacy Access to, use of and satisfaction with services Enrollment, repetition, dropout rates; share of schools with active parents organizations Intermediate indicators OUTPUTS INPUTS Goods and services generated Number of schools built, textbooks, etc. Financial and physical indicators of resources provided Spending on primary education 6

10 Draft for Comments. April, 2001 Table 1 illustrates goals and some of their corresponding intermediate and final indicators. Table 1. Examples of Final and Intermediate Indicators Intermediate indicator Goal (input and output) Reduce extreme poverty and expand economic opportunities for the poor. Enhance the capabilities of poor men and women. Expenditure on infrastructure Expenditure on and number of beneficiaries of job training programs Percentage of roads in good and fair condition Expenditure on primary education as a share of national income. Expenditure on primary health care as a share of national income. Percentage of schools in good physical condition Pupil-teacher ratio Number of doctors per 100,000 inhabitants Reduce the vulnerability of the poor. Expenditure on safety net programs Percentage of poor households/individuals receiving transfers from the government Final indicator (outcome and impact) Incidence of extreme poverty: percentage of population whose consumption falls below the poverty line Poverty gap ratio Income/expenditure of the poorest 20% of the population as a share of the total income/expenditure of the whole population Unemployment/underemployme nt rate Percentage of the poor population with access to microcredit programs Literacy rates Learning achievement Dropout and repetition rates Net enrollment in primary education Percentage of population below the poverty line with access to health care facilities Infant, child, and under-five mortality rate Maternal mortality rate Malnutrition rate Variability of household consumption Percentage of AIDS orphans protected Although the main objective of the monitoring system is to track progress in poverty outcomes and impacts, both final (outcome and impact) and intermediate indicators (input and output) should be tracked. 2 Monitoring final indicators helps to judge progress toward the goals set. But final indicators are the result of several factors, many of which are outside the control of policymakers and program administrators. Intermediate indicators, on the other hand, generally change as a result of actions by the government and other agents. Moreover, final indicators generally change slowly over time, while intermediate indicators change more rapidly, giving an indication if not on what is happening with well-being at least what is happening with some of its determinants. This can make it possible to take corrective action while a program is being 2 In this respect a poverty monitoring system combines implementation monitoring and performance- or results-based monitoring (sometimes the term poverty monitoring system is also used to refer to outcome/impact monitoring only). 7

11 Draft for Comments. April, 2001 implemented. Finally, information on intermediate indicators is often easier to collect (we will return to this point below when discussing sources of data). The most useful intermediate indicators are those that refer to key determinants of impact or outcome and that vary across areas or groups or over time. So, for example, in a country where all schools have more or less the same teacher-to-student ratio, the teacher-to-student ratio would not be a very useful intermediate indicator to monitor differences in quality of education across regions (although it could still be useful to monitor changes over time). Final and intermediate indicators should be complemented with other selected indicators to measure overall country performance and account for the context in which the poverty reduction strategy is being implemented. For example, indicators measuring exogenous factors that are likely to impinge on outcome indicators such as rainfall or external demand for a country's goods should be included in the monitoring system. In general, good indicators share a number of features. These are summarized in Box 2. Box 2. Features of Good Indicators A good indicator: Is a direct and unambiguous measure of progress more (or less) is unmistakably better. Is relevant it measures factors that reflect the objectives. Varies across areas, groups, over time, and is sensitive to changes in policies, programs, institutions. Is not easily blown off course by unrelated developments and cannot be easily manipulated to show achievement where none exists. Can be tracked (better if already available), is available frequently, and is not too costly to track. The choice of indicators is clearly dependent on the types of data that are available in a country, as well as on what can be feasibly monitored given resource and capacity constraints; in fact, the process of selecting indicators should start from an analysis of what is available and what is feasible, and indicators that are not yet available should be included in the monitoring system only if it is realistic to set up a mechanism to collect and analyze data on such indicators. To see the intermediate and final indicators that have been selected in practice, look at Case studies 1 and 2, which provide examples of the indicators used to monitor the effectiveness of the poverty reduction strategy in Uganda and Tanzania. 1.3 Disaggregating indicators The decision on the level of disaggregation of indicators is as important as the choice of indicators itself. These are in a sense joint decisions that are usually considered at the outset, based on existing data sources and on the goals that a strategy aims to achieve. Indicators can be disaggregated along various dimensions, including location, gender, income level, and social group (based on ethnicity, religion, tribe, caste). Aggregate, country-level indicators are useful 8

12 Draft for Comments. April, 2001 as they give an overall picture of where a country stands in comparison with others. However, aggregate indicators tend to mask significant differences across areas, gender, or social groups and it is hard to design good policies and programs to reduce poverty without a disaggregated picture that captures these differences. The appropriate type and level of disaggregation depend on country conditions and the indicator itself. Here are some examples. A basic type of disaggregation is by geographical areas including urban/rural, administrative units and geo-climatic zones. Calculating disaggregated urban and rural indicators is common, and essential, but not always sufficient. Smaller cities often tend to be more similar to rural areas than to megacities, for example, in terms of the importance of agriculture as a source of livelihood. So it may be useful to disaggregate further among urban areas by size of settlement or at least to distinguish megacities from the rest. Similarly, the capital city often tends to have different characteristics: higher average income, better availability of services, a larger share of employment in services, and so on. Thus, it may be useful to construct separate indicators for the capital. Most countries are divided into administrative units states, regions, provinces, districts, municipalities and villages, and so on and these can be used as a basis of disaggregation. Ideally, there would be indicators for each administrative level with decision-making power over resources, or to which resources are allocated. In practice, however, the availability of data and resource constraints will determine the lowest level of disaggregation that is feasible. A third type of geographical disaggregation is by geo-climatic zones. Most countries have a number of geographical zones characterized by different soils, rainfall, topography, and consequently different agricultural practices, settlement patterns, ease of access, and so on. Another basic type of disaggregation is by gender. Appropriate gender indicators measure factors that vary by gender and take into account the impact of biological differences. For example, life expectancy tends to be higher for women, so a lower life expectancy for women than for men is usually an indication that women may be suffering severe health risks at childbirth. See the Gender chapter for more information. Disaggregating by income, consumption, or asset ownership level is a common way to see how indicators vary across the population. It is usually preferable to a simple poor nonpoor disaggregation, as it captures the fact that many household and individual characteristics vary along a continuum. There are often significant differences among those classified as poor, and those just below the poverty line generally have very similar characteristics to those just above it. So it is desirable to divide the population in groups of equal size rather than simply into poor and nonpoor. Some commonly used groupings based on income and consumption level are: Name Number of groups Share of the population Deciles percent Quintiles 5 20 percent Quartiles 4 25 percent n-th percentile N 100/n percent 9

13 Draft for Comments. April, 2001 Dissagregating indicators by, for example, quintiles is important to monitor whether improvements reach the worse-off as well as the better-off. Nation-wide average targets, such as those of the International Development Goals, can often be reached with different degrees of improvement for different groups; 3 if improving the well-being of the poorest is important, then tracking indicators disaggregated by quintile is essential. In most countries there are significant differences across socially defined groups, whether along ethnic, tribal, religious, or other lines. The definition of the relevant groups will naturally vary across countries. Finally, it is important to recognize that disaggregating indicators by areas, groups, and the like usually has political consequences and must be done carefully. Furthermore, monitoring indicators disaggregated by administrative area almost always requires complementary efforts to build capacity for monitoring and analysis in the decentralized administrative units, a point highlighted in Case Study 1 on Uganda. 1.4 Setting targets Once indicators are selected, it is useful to assess baseline values and set quantitative targets for at least some of them. Baseline values can be obtained from existing data, if they are of reasonable quality and not too old. 4 Where data for an indicator do not yet exist, the first available estimate, if it comes within a reasonable amount of time, or a preliminary estimate subject to revisions, can be used as the baseline. Setting targets is a complex task; we offer some general guidelines here, and additional guidance on the technical aspects of setting targets for different indicators, in the chapter on Informed Target Setting. First, targets should be selected on the basis of the current situation and what is attainable in a given country in a given time. Even if a country chooses goals consistent with the IDGs (see Box 1), the indicators and targets selected may not be the same. The target of achieving universal primary school enrollment obviously is not relevant for a country where this has already been achieved. Second, targets may be set at different levels of disaggregation. In addition to national-level targets, specific targets can be set for certain regions or groups. For example, for most countries, educational targets are not very useful unless they are differentiated by gender, and for large countries such as Brazil and India, geographical targets make good sense. Third, including qualitative and subjective factors in goal setting is important. Many factors that affect quality of life cannot be easily quantified but are not for this reason less important. Where feasible, qualitative and subjective indicators could be added for example, whether or not people perceive themselves as being poor. Setting quantitative targets may not make sense for such indicators. 3 For a discussion of how health targets can be reached with different degrees of improvement for the poorest and richest, see Davidson R. Gwatkin (2000a and b). 4 For example, existing household survey data may be too old, or the sampling methodology may not be such as to ensure representativeness. 10

14 Draft for Comments. April, 2001 Fourth, as a general rule, improvements become more difficult as levels improve. For example, it is in general more difficult to reduce income poverty from 10 percent to zero than from 40 percent to 30 percent, because the target group generally becomes more difficult to reach. Fifth, if a particular indicator has continuously worsened in the recent past, it may not be realistic to set a target indicating a substantial improvement in the short term. Most likely, it will take some time for that indicator to stabilize and start improving. Finally, it is essential to consider the resource implications of the selected targets and their feasibility. Resources may have to be shifted from some sectors and programs toward activities that are in line with the selected targets. See the chapter on Informed Target Setting for a more detailed description of the costing of targets. Figure 2 summarizes the steps involved in selecting indicators and setting targets, and points to documents providing guidance on each step. Figure 2: Selecting indicators and setting targets Are there agreed indicators and targets for the poverty reduction strategy? YES NO Are there agreed short-term and long-term indicators? - Long-term impact indicators (frequency: three to five years) - Medium- and short-term outcome indicators (frequency: annual or more) - Indicators of inputs and outputs to monitor public actions (frequency: quarterly or more) - Indicators at the right level of geographical and social disaggregation - Gender-sensitive indicators YES NO Discuss indicators at a national forum Seek technical support from donors Resources: this chapter; Participation chapter Resource: I-PRSPs/PRSPs prepared in other countries, Resource: International Development Goals, Are there agreed targets? - Targets should be ambitious but achievable YES NO Check international experience Study evolution of indicators over time Resource: chapter on Informed Target Setting; international databases; World Development Indicators NEXT STEP: Poverty Monitoring System 11

15 Draft for Comments. April, Determining data requirements As mentioned, both intermediate and final indicators should be tracked. So a good poverty monitoring system would include data on both categories of indicators. These would be collected through a number of different instruments and by different agencies. This last point is important: the fact that a good poverty monitoring system requires data on different indicators does not mean that one agency needs to be in charge of all data collection this would be neither desirable nor efficient. Data on intermediate indicators are usually collected by the Treasury or Finance Ministry and sectoral ministries at the central and local level through financial and management information systems. These systems collect data on public expenditures in various sectors and on activities and outputs produced by such expenditures. For example, the Treasury or Finance Ministry will collect data on expenditures in education, while the Ministry of Education will have data on schools built, textbooks purchased, scholarships provided, training activities and so on. Data from administrative records usually exist in countries, although there may be problems with their accuracy, timeliness and comprehensiveness. Data on the number of staff in key sectors come from sectoral ministries or the ministry in charge of public administration. Information on outcome and impact indicators normally needs to be collected from beneficiaries through household or individual surveys and participatory methods. Because of the need to collect information directly from households and individuals, outcome and impact data are costlier to collect and require more time. Particular attention is needed to obtain reliable information from women and possibly other groups such as children, the elderly, or excluded minorities, that may not be easily reached or may not feel comfortable responding to interviewers. Why is it necessary to collect data on access to and use of services from households, in addition to using data from administrative records? Why, for example, survey households are needed to know how many children attend school rather than use only Management Information Systems data from the Ministry of Education on enrollment rates? First, data collected from households are more reliable: households have less incentives to report school attendance incorrectly than program administrators and local officials, whose budget allocations and incentives may depend on achieving enrollment targets. Second, household surveys and participatory studies generally collect other information from households, such as income or consumption, education status of the parents and employment status, or reasons not to attend school; this additional information makes it possible to analyze the causes of trends in enrollment rates. This is not to say that MIS data on use of services are not useful, only that they should be checked against and complemented by information collected directly from households. A good monitoring system should include also data on external factors that may influence the effectiveness of the poverty reduction strategy, such as weather or external market factors. Table 2 summarizes collection instruments, agencies usually responsible and level for different indicators. 12

16 Draft for Comments. April, 2001 Table 2. Data for Monitoring and Sources Type Indicator Instrument Agency Level Input Public finance data: revenues, expenditures by category Budget documents; actual expenditure data Output Outcome Outcome/ Impact Other Other Human resources Outputs of public expenditures: infrastructure, services provided Access to, use of and satisfaction with services Household consumption and income; living conditions; social indicators; household priorities; perceptions of well-being National accounts: Gross domestic product, consumption, investment, exports, imports, etc. Consumer and producer prices Climatic data: temperature, rainfall, water flows, etc. Expenditure tracking surveys Payroll data Administrative systems, Management Information Systems Community surveys Priority and quick monitoring surveys; multitopic household surveys; qualitative studies Household budget/ expenditure/ income surveys; single-topic surveys (for example labor force surveys); multi-topic household surveys (such as Living Standard Measurement Surveys and Demographic Health Surveys); qualitative studies System of national accounts, trade statistics Direct measurement Ministries of finance and planning and public administration; sectoral ministries; public accounting and auditing agencies Sectoral ministries; project implementation units; local administrations and local service providers Central statistical agency; local service providers; others Central statistical agency Central statistical agency; central bank National weather agency; others National and various sub-national administrative levels National and various sub-national administrative levels; facilities (schools, clinics, etc.) Households and individuals; facilities (schools, clinics, etc.); communities Households and individuals; communities National (largest subnational levels in some cases) As detailed as possible For a more detailed discussion of various data collection instruments, see the chapters on Poverty Data and Measurement and Building Statistical Capacity. Note that data from these various sources are complementary, not substitutes for one another. Having very good household-level data on consumption and incomes will not be sufficient to understand trends in poverty outcomes; accurate and timely data on public expenditures and public services are needed as well The increased attention that poverty reduction strategies place on final indicators should not reduce attention to intermediate indicators, or shift resources away from tracking them. 13

17 Draft for Comments. April, Determining the frequency of monitoring The decision on how frequently a given indicator needs to be monitored depends on a careful assessment of the tradeoff between the desirability of recent data and the cost of collection, much like the decisions on which indicators to track and at what level of disaggregation. Data on input indicators, such as public expenditures, are tracked at least annually, and in most cases more often (monthly, or quarterly), as part of budget tracking mechanisms. Data on outputs are most often available on an annual basis, but it is highly desirable to have information on key outputs mid-way through the budget year to inform mid-course corrections and decisions on budget allocations for the following year. Data on some outcome indicators should also be available annually. Data on impacts, on the other hand, are usually not available annually, both because it is costly to collect and analyze household survey and participatory data, and because impact indicators do not usually change rapidly. Table 3 indicates the desirable frequency of collection for the various indicators listed in the previous table. 5 Table 3. Frequency of Data Collection Type Indicator Instrument Frequency Input Public finance data: revenues, expenditures by category Budget documents; actual expenditure data Expenditure tracking surveys Monthly or quarterly where possible; at least yearly Output Outcome Outcome/ Impact Other Other Human resources Outputs of public expenditures: infrastructure, services provided Access to, use of and satisfaction with services Household consumption and income; living conditions; social indicators; household priorities; perceptions of well-being National accounts: Gross domestic product, consumption, investment, exports, imports, etc. Consumer and producer prices Climatic data: temperature, rainfall, water flows, etc. Payroll data Administrative systems, Management Information Systems Community surveys Priority and quick monitoring surveys; multitopic household surveys; qualitative studies Household budget/ expenditure/ income surveys; multitopic household surveys; qualitative studies System of national accounts, trade statistics Direct measurement Possibly every six months; at least yearly Yearly where possible Every three to five years Monthly or quarterly where possible (trade statistics, for example); at least yearly Monthly or quarterly price collection; consumer price index basket updated at least every five years Daily where possible 5 Guidance on the frequency of collection of gender-based indicators can be found in the chapter on Gender. 14

18 Draft for Comments. April, Elements of poverty monitoring systems that often need attention Most countries already have monitoring systems in place to track most, if not all, the indicators needed to monitor the effectiveness of poverty reduction strategies. So what more needs to be done? Recent experience in countries that are developing and implementing poverty reduction strategies points to the need to devote attention early on to some key elements of the system. Frequent problems in tracking intermediate indicators are: Actual expenditure data are not timely. In many countries actual expenditure data are available only with a significant time lag. This is less problematic for recurrent expenditures (especially salary but also non-salary), where actual expenditures are often fairly close to budgeted amounts, but can seriously limit a country s ability to track capital expenditures, that are often quite different from budgeted amounts. Programs to improve expenditure tracking at the central and decentralized levels for example through the establishment of well-designed reporting formats and computerization can improve the timeliness of expenditure data. 6 Input data (expenditures and human resources) cannot be easily related to outputs, so it is hard to estimate the cost of providing services. For example, a large share of expenditures in education is for general administration, and it is not clear how much of this supports primary versus secondary or tertiary education. So the cost of providing for example a year of schooling to a primary school child cannot be estimated accurately. Solving this problem requires moving towards activity-based costing, where all expenditures are related to specific activities and outputs. This is done extensively only in a small number of countries, but in most countries there is scope to move in this direction. 7 Disaggregated spending data are unavailable or inaccurate. Without data disaggregated at the level of the facilities or agencies that provide services, it is hard to assess whether public funds reach the facilities or not. Where local government accounts are not available or are of poor quality, expenditure tracking surveys can be conducted. In Uganda, spending data for collected from a random sample of public schools revealed that less than 30 percent of the funds intended for non-salary public spending actually reached schools because district administrations kept and used the rest of the funds. This finding led to the decision to inform the public on allocations and to changes in spending procedures. The survey instruments and methodologies used are available and can be applied elsewhere. 8 In tracking outcomes and impact, other issues have emerged. It takes a long time to process data from household surveys and make them available for analysis. Data entry, cleaning and organization often take years. This need not be: there are ways to shorten the process considerably. For example, data entry can be carried out in the field or in decentralized field offices concurrently with data collection; there are even experiments to eliminate paper questionnaires completely and enter data directly on disk. Data cleaning can be speeded up considerably by using pre-coded questionnaires and data 6 For more discussion of systems to improve the tracking of public expenditures, see the chapter on Public Spending. See also the assessment of expenditure tracking systems done by the World Bank for the Highly Indebted Poor Countries initiative: 7 For more information on costing programs, see the chapter on Informed Target Setting. 8 For more detail on the methodology and findings, see Abdo and Reinikka, 1998, and Republic of Uganda, Survey instruments can be found at tools/tools.htm. 15

19 Draft for Comments. April, 2001 entry programs that identify entry errors and inconsistencies between variables (for example a mother who is younger than one of her children). Moreover, when data entry takes place while in the field, errors can be corrected through recall or re-interviewing. 9 There is a need to introduce quick monitoring tools to gather information from households on an annual (or more frequent) basis. Even when data from household surveys are processed and made available quickly, it remains the case that these surveys take time to conduct (especially if data are collected over the course of a year to capture seasonal patterns) and are costly, so it may not be affordable to conduct them every year. How can changes in household and individual well-being be tracked more frequently? There are now quick monitoring tools that have been tested in different countries and can be applied fairly easily the Core Welfare Indicators Questionnaire is a good example (see Box 3). Other examples are the citizen scorecards piloted in Bangalore, India (see Case Study 3) and the user surveys piloted in Uganda, that complemented the expenditure tracking surveys cited above. 10 Box 3: The Core Welfare Indicators Questionnaire (CWIQ) A number of countries in Africa (for example Ghana and Tanzania) have started using a new survey tool, the Core Welfare Indicators Questionnaire (CWIQ) for monitoring outputs and outcomes in the context of poverty reduction strategies. The Core Welfare Indicators Questionnaire (CWIQ) is a household survey designed to provide very rapid feedback through the tracking of leading indicators, and can show who is and who is not benefiting from programs and policies. It focuses on simple indicators of usage, access and satisfaction. The CWIQ is a ready-made survey package which national statistical offices can implement on an annual basis and can supplement, when necessary, with special modules. It is meant to complement other surveys. It is designed to be administered to large samples of households so that results can be disaggregated to relatively low levels and to be repeated annually, so that time-series can be quickly built up. The standard output tables and graphs present access, usage and satisfaction indicators broken down by geographic and socio-economic groupings. The CWIQ does not collect information on consumption or income, which cannot be done accurately using a short questionnaire, but can collect information on indicators that are related to economic well-being, such as consumption of certain goods or ownership of assets. A recent multi-topic or budget survey is usually used to identify core indicators that are easy to monitor and correlated with consumption or income; if such a survey is not available, information from a participatory poverty assessment can be used, as was done for the first pilot in Ghana. The CWIQ can include up to ten such indicators, and these can be used as proxy indicators to track changes in consumption/income and income poverty. 2. Designing Impact Evaluations Poverty monitoring provides crucial information to assess overall progress in achieving poverty reduction goals and understand changes over time and space. However, complementary tools such as impact evaluations are required to inform policy makers and the public on which public actions have been effective and which ones have not worked so well in reducing poverty. An impact evaluation assesses the changes in well-being that can be attributed to a particular 9 For more information on ways to improve the timeliness of household survey data, see Grosh and Munoz, For more information on the Core Welfare Indicators Questionnaire, see www4.worldbank.org/afr/stats/cwiq.cfm; copies of the brochure, questionnaire, handbook, and various other documents about the CWIQ can be downloaded from the site. For more information on user surveys in Uganda, see tools/tools.htm. 16

20 Draft for Comments. April, 2001 Section 3: Challenges Ahead for Monitoring and Evaluation 3.1 Assessing the process of formulation and implementation of poverty reduction strategies The main objective of a poverty reduction strategy is to reduce poverty, and this chapter has focused on monitoring progress in achieving poverty reduction goals and evaluating the poverty impact of interventions that are part of the strategy. But the process of formulating and implementing a poverty reduction strategy also seeks to achieve several objectives: increase country ownership; foster partnership between the government and civil society on one hand, through deeper participation, and the government and donors on the other hand; take a longterm, comprehensive approach to poverty reduction. It would be important to monitor these objectives and assess whether they are met. The steps described in the first section to set up a poverty monitoring system apply equally to setting up a system to monitor progress towards process objectives. Agreement is needed on the objectives to achieve, and on the indicators to be used. Objectives and indicators should be selected in a participatory way. Indicators could refer to inputs and outputs of the process, as well as to outcomes; for example, the following indicators have been suggested to monitor participation in the preparation of a Poverty Reduction Strategy Paper: 12 Input: public resources used to increase quality and scope of participation Output: measures of the extent to which meaningful participatory arenas (that include all stakeholders who want to participate) have been opened across the country to discuss the design, implementation and M&E of a PRSP Outcome: measures of the extent to which the PRSP takes into account the needs and priorities of key stakeholders including poor people; civil society and government have a higher capacity to decide on the country s poverty reduction strategy and more opportunities to negotiate with donors and creditors over it. Indicators should be disaggregated by gender, geographic area, social group, etc. where appropriate (for example, the number of participatory meetings held could be disaggregated by area; participation of women could be tracked separately) and should be specified precisely whenever possible. Where data exist on those indicators that can be quantified, it may be useful to identify initial ( baseline ) values and define targets. For example, baseline values for participation indicators could reflect the situation before the PRSP process is initiated. Where data do not exist, as will often be the case with process indicators, a system to collect the needed data and analyze them would have to be set in place. As for indicators in general, what is desirable may not be feasible or affordable, so the final decision on what to monitor, with what instruments and what frequency will be influenced by resources available. In many cases also, process indicators may be qualitative in nature and not lend themselves to be quantified. 12 Adapted from a presentation by Rosemary McGee and John Gaventa of the Institute for Development Studies. 25

21 Draft for Comments. April, 2001 The process of selecting indicators and monitoring the process of formulating a poverty reduction strategy offers a real opportunity to foster partnership between the government, civil society organizations, and donors. It also is a learning opportunity, as most of the experience so far in assessing process objectives has been gained at the micro level (projects and programs) rather than at the macro level (strategy). 3.2 Evaluating the overall poverty impact of poverty reduction strategies After a few years of implementation of a poverty reduction strategy, the question of whether the strategy as a whole (rather than specific interventions within it) has been effective in reducing poverty may arise. Evaluating the poverty impact of the entire strategy poses a tremendous challenge as it requires an evaluation framework that considers a large number of economic and institutional changes occurring simultaneously and can sort out the causal relationships between actions. One possible approach is to use methodologies similar to those for evaluating the poverty impact of country-wide, or full-coverage interventions: comparing the situation before and after implementation of the strategy using time series (see reflexive comparison in Technical Note 2); simulating the situation without the strategy using computable general equilibrium (CGE) models; and comparing countries with different strategies through regression analysis and other methods (see Technical Note 3). For indicators of poverty that capture empowerment and security dimensions, participatory methods may be more appropriate. Experience is limited and much remains to be learned. Because of the complexity of such overall evaluation exercises and the capacity and resources they require, countries are not expected to carry them out. Moreover, given that the poverty impacts of a strategy may only be observed several years after the start of implementation as noted, it takes time for policies and programs to affect well-being it is not advisable to evaluate the overall poverty impact of a poverty reduction strategy within the three-year time frame of a Poverty Reduction Strategy Paper. Within this time frame, it is possible to assess the process of formulating and implementing the strategy (as discussed in the previous section), monitor outcomes, and carry out other types of evaluation including qualitative and participatory assessments that examine the links between the inputs and processes of the strategy and any outcomes observable within the three-year time frame (see Technical Note 1). What is most important in the short- and medium-term is to set up a solid monitoring system: without the basic information collected through the monitoring system, no evaluation exercise can be carried out. 4. Strengthening M&E capacity and feedback mechanisms 4.1 Strengthening capacity Poverty monitoring and impact evaluation activities involve the participation of several agencies both inside and outside the government, each with their role. Within the government, central ministries such as finance and planning usually have a major role in designing the overall monitoring and evaluation strategy, monitoring its implementation, and using the results, as well as providing key data on expenditures; sectoral ministries usually provide data on outputs; the central statistical agency is usually responsible for the collection of data from households and individuals. Agencies and institutions outside the government, such as research centers, 26

22 Draft for Comments. April, 2001 universities, and NGOs often also collect and analyze information. Donors can provide technical assistance to strengthen capacity. These roles are summarized in Box 6. Box 6. Roles of various agencies in monitoring and evaluation Central ministries such as planning and finance are usually in a good position to coordinate the design, monitoring, and support for M&E activities. The finance ministry also provides key data on public expenditures. Line ministries are usually in charge of sectoral program coordination and supervision. Thus they play an important role in supervising the implementation of monitoring and evaluation activities at the sectoral level, and they are the key source of administrative records and data from Management Information Systems Project implementation agencies are in charge of project and program management. They are responsible for the timely and appropriate implementation of program monitoring and evaluation. Central statistical offices are key providers of data as well as of expertise in data collection and analysis. Universities, research centers, and consulting firms are potential suppliers of analysis and evaluation skills and also can offer training in a range of skills. Development assistance agencies can help develop monitoring and evaluation capacity by providing technical assistance. Strong country demand at all levels is generally the main precondition for the development of a national monitoring and evaluation system. Sustainable capacity is usually built up if governments and civil society are truly committed to measuring the outcomes and impact of public action and to using this information to achieve better results. Thus, the participatory processes followed in designing poverty reduction strategies can be critical in creating a strong demand for monitoring and evaluation. Donors can contribute to create demand for M&E activities through the requirements of their assistance. For example, one of the conditions associated with the provision of concessional assistance and debt relief by the IMF and World Bank under the Poverty Reduction Strategy Papers approach is that governments prepare an annual progress report on the implementation of the poverty reduction strategy. This annual report would discuss actions taken and changes in those indicators that are tracked annually; if annual targets were set, the report would discuss whether they were attained and indicate the reasons for any differences between actual values and targets. 13 While such donor requirements do create demand for M&E, sustainable capacity will be built only if there is strong in-country demand. Once there is a strong country demand for monitoring and evaluation, feasible options to build capacity vary across countries depending on local circumstances and opportunities, the actors involved, the institutional framework, and the distribution of existing capacity across agencies. 14 An important consideration is that it may be appropriate to scale up monitoring and evaluation activities gradually. Experience suggests that it may be better to put in place a few mechanisms that can be implemented immediately than to start with the design and development of a 13 The annual progress report would also discuss any modifications in the strategy or its implementation that may be necessary given the findings of monitoring and evaluation activities. See the December 1999 Joint Bank-Fund Board paper on PRSPs Operational Issues. 14 See for example Blank and Grosh, 1999, on how to use household surveys to build analytical capacity. 27

23 Draft for Comments. April, 2001 comprehensive or very sophisticated setup. A first step can be to take stock of existing monitoring and evaluation capabilities and activities among central and line ministries, local governments, national statistical agencies, and other organizations such as universities and NGOs. On the basis of this assessment, various alternatives can be implemented to ease capacity constraints and develop local skills, including: Establish partnerships to collect and analyze data, as well as to provide training on skills relevant to monitoring and evaluation. Potential partners are universities, research institutions, NGOs, and consulting firms as well as development agencies. Collaboration with these institutions can take several forms, including carrying out joint evaluations, providing grants for the professional development of monitoring and evaluation specialists, and contracting out survey implementation. Disseminate national and international lessons about experience in monitoring and evaluation. Identify good-practice examples within the country and in similar countries and create a database. Selected cases from this database can be presented at workshops for key central and local government officials. Build a network to facilitate exchange among practitioners, academics, and civil servants in charge of monitoring and evaluation activities. Network activities can include knowledge dissemination and training. At the international level, the International Development Evaluation Association (IDEAS) provides a forum to exchange information on good practices and methodologies. As decentralization of administrative functions and service provision takes place in a country, it is important to build up M&E capacity at the sub-national level. Regional and provincial administrations, and citizens, will need to assess the effectiveness of the strategy pursued at the local level. Central statistical agencies are reluctant at times to build decentralized capacity, but this reluctance can be overcome if central and local M&E systems are seen as complementary. National agencies can continue to have responsibility for the conduct of data collection and analysis exercises at the national level; local agencies can develop the capacity to analyze subsets of the national data as well as collect and analyze data to assess the impact of local policies and programs. The chapter on Building Statistical Capacity discusses in more detail how to assess capacity and develop short- and long-term plans to strengthen capacity for quantitative data collection, while section 5 below discusses the role of non-governmental actors See also for additional information on assessment tools and lessons learned in building institutional capacity for monitoring and evaluation. 28

24 Draft for Comments. April, Strengthening feedback mechanisms Monitoring and impact evaluation should not be stand-alone, technical activities. They should be closely linked to decision-making processes at all levels and provide feedback to project managers, policymakers, and civil society on, among other things, the performance of existing policies and programs. Thus, a crucial element of the monitoring and evaluation system is the existence of a feedback process. A feedback process is a mechanism by which monitoring and evaluation results are disseminated and used to decide on future courses of action. Results should be disseminated broadly. Monitoring and evaluation systems that provide results to only a selected group of users (central ministries, for example) risk being underutilized and losing financial and political support. Wide dissemination of results reinforces the system by strengthening an outcome-based culture. The dissemination strategy should accommodate the diverse information needs of different groups including policymakers, program managers, program beneficiaries, the general public, the media, and academics. For example, reports that include main findings and emphasize implications for policy and program design can be distributed among government officials in central and line ministries, as well as in local administrations. Detailed reports can be produced for program administrators and researchers. Press releases can be used to reach the media. Workshops and seminars can be used to disseminate results among the general public, and civil organizations. Posting of information on the web, if possible, makes it available to interested audiences within and outside the country. It is important that findings and recommendations be accessible to community councils, local women s organizations, and ethnic, religious, environmental, and other groups representing communities to whom programs are targeted. Most of these groups may not have access to information technology and conventional dissemination mechanisms. In these cases, alternative dissemination methods such as meetings, pamphlets, posters, and so on may be required. Dissemination materials prepared in more than one language and separate meetings with different groups (for example, men and women) may also be required. Active participation of NGOs and other local organizations may be crucial to ensure that all sectors of the community are reached. In addition to results, the actual data as well as careful documentation of methods of analysis should also be made available to the public. Reluctance in releasing unit record data can give rise to suspicion, while open access and discussion over data, methods and results foster transparency and broad acceptance of the findings. Open access to unit record data also enables non-governmental organizations to carry out independent analysis, and increases demand for data, which helps ensure the sustainability of the M&E system. In some countries there are legal impediments to the dissemination of raw data, related to the protection of privacy; these can be overcome with technical solutions that make it very hard to identify respondents and changes in the legal framework; many countries now grant open data access, and lessons have been learned from their experience. Beyond broad dissemination, a well-established process to feed M&E results back to policymakers is crucial if results are to be used in formulating policy. Since key policy decisions 29

25 Draft for Comments. April, 2001 are made at the time of budget formulation, key results should be available then. This means, in particular, that data for the first six months of the fiscal year should be available not just on expenditures but also on outputs. Any data on other intermediate and final indicators tracked annually should also be made available at the time of budget formulation. In some countries, poverty monitoring units have been set up with the explicit purpose of providing policymakers with information to base decisions on. These units have been most successful where they have been located close to decision-making centers (such as for example in the Prime Minister s Office) and when they have acquired adequate capacity so as to provide competent and timely information. In other cases, independent agencies have been set up (such as the Observatoires in some West African countries). 5. Promoting participation in monitoring and evaluation Non-governmental actors, from researchers to community organizers to representatives of the poor have an important role to play in monitoring and evaluation: they can contribute their knowledge and expertise to the design of the M&E system; carry out M&E activities directly; and use the results to keep governments honest. Broad consultations during the design of the monitoring and evaluation system are important to build consensus on what to monitor and what to evaluate the selection of indicators and targets and generate a sense of ownership among different groups in society, thus increasing the acceptance and use of findings. Consultations help identify adequate indicators of people s perception of well-being, and bring into the process the expertise of non-governmental organizations. In addition to providing their views, expertise and knowledge during the design of the system, civil society organizations can contribute directly to implementing M&E activities, either independently or under contracts from the public sector. Research organizations and universities often have the capacity and expertise to carry out surveys and participatory work and analyze the results, while interest groups and community-based groups can take advantage of easy access to their members to get their views and opinions. Also, civil society organizations are sometimes more experienced than government agencies in the use of participatory methods of data collection and analysis. Finally civil society organizations have a crucial role to play as users of M&E results. Wide dissemination of results encourages participation. By accessing monitoring and evaluation findings, civil society organizations can generate a participatory review process of poverty reduction efforts that increases accountability and transparency of public resource allocation and public actions. The chapter on Participation expands on these issues and discusses alternative strategies to promote participation depending on country circumstances. For information on promoting women s participation, see the chapter on Gender. 30

26 Draft for Comments. April, 2001 Guide to Resources Baker, Judy Evaluating the Poverty Impact of Projects: A Handbook for Practitioners. Directions in Development, World Bank, Washington, D.C. This handbook seeks to provide project managers and policy analysts with the tools needed for evaluating the impact of interventions. It includes a discussion of evaluation methodologies and implementation issues and presents several case studies, some of them also included in this chapter. Available at MacKay, Keith Evaluation Capacity Development: A Diagnostic Guide and Action Framework. ECD Working Paper Series 6. World Bank, Operations Evaluation Department, Washington, D.C. Available at This guide provides a detailed checklist of issues to be considered in developing a country s evaluation capacity. Available at World Bank CWIQ (Core Welfare Indicators Survey) Handbook and CD-Rom. Africa Operational Quality and Knowledge Services, World Bank, Washington, D.C. This handbook provides guidance on the use and implementation of the CWIQ. Available at: Websites Monitoring and Evaluation Capacity Development ( ml) contains assessment tools and lessons learned in building institutional capacity for monitoring and evaluation. PovertyNet ( provides a number of resources for poverty monitoring including links to the poverty monitoring database, LSMS site, Poverty in Africa site, Africa Household Survey Databank and impact evaluation site. Poverty Reduction Strategy Papers ( includes interim and final PRSPs prepared by countries. 31

27 Draft for Comments. April, 2001 Bibliography Abdo, Emmanuel, and Ritva Reinikka Do Budgets Really Matter? Evidence from Public Spending on Education and Health in Uganda. Policy Research Working Paper World Bank, Washington, D.C. Baker, Judy Evaluating the Poverty Impact of Projects: A Handbook for Practitioners. Directions in Development, World Bank, Washington, D.C. Blank, J., and Margaret Grosh "Using Household Surveys to Build Analytic Capacity." World Bank Research Observer 14 (2): Carvalho, Soniya, and Howard White Combining the Quantitative and Qualitative Approaches to Poverty Measurement and Analysis: The Practice and the Potential. Technical Paper 366. Washington, D.C.: World Bank Ezemenari, Kene, Gloria Rubio, Anders Rudqvist, and K. Subbarao. Forthcoming. Impact Evaluation: A Position Paper. World Bank, Poverty Reduction and Economic Management Network, Poverty Division, Washington, D.C. Fuller, Bruce, and Magdalena Rivarola Nicaragua s Experiment to Decentralize Schools: Views of Parents, Teachers, and Directors. Working Paper Series on Impact Evaluation of Education Reforms. 5. World Bank, Development Economics Research Group, Washington, D.C. Grosh, Margaret, and Juan Munoz A Manual for Planning and Implementing the LSMS Survey, LSMS Working Papers Series 126 (available also in Russian and Spanish). Grossman, Jean Baldwin Evaluating Social Policies: Principles and U.S. Experience. World Bank Research Observer 9 (2): Gwatkin, Davidson R. 2000a. Health Inequalities and the Health of the Poor: What Do We Know? What Can We Do? WHO Bulletin, January Gwatkin, Davidson R. 2000b. Meeting the 2015 International Development Target for Infant Mortality: How Much Would the Poor Benefit? World Bank, Washington, D.C. June Hentschel, Jesko Distinguishing between Types of Data and Methods of Collecting Them. Policy Research Working Paper World Bank, Poverty Reduction and Economic Management Network, Poverty Division, Washington, D.C. King, Elizabeth, and Berk Ozler What s Decentralization Got to Do with Learning? The Case of Nicaragua s School Autonomy Reform. Working Paper Series on Impact Evaluation of Education Reforms 9. World Bank, Development Economics Research Group, Washington, D.C. Kozel, Valerie, and Barbara Parker Poverty in Rural India: The Contribution of Qualitative Research in Poverty Analysis. World Bank, Poverty Reduction and Economic Management Sector Unit, Washington, D.C. Mackay, Keith Evaluation Capacity Development: A Diagnostic Guide and Action Framework. ECD Working Paper Series 6. World Bank, Operations Evaluation Department, Washington, D.C. 32

28 Draft for Comments. April, 2001 MacKay, Keith, and Sulley Gariba (eds.) The Role of Civil Society in Assessing Public Sector Performance in Ghana: Proceedings of a Workshop. World Bank, Evaluation Capacity Development, Operations Evaluation Department, Washington, D.C. Narayan, Deepa Participatory Evaluation: Tools for Managing Change in Water and Sanitation. Technical Paper 207. Washington, D.C.: World Bank. Nicaragua Reform Evaluation Team Nicaragua s School Autonomy Reform: A First Look. Working Paper Series on Impact Evaluation of Education Reforms 1. World Bank, Poverty and Human Resources Division, Policy Research Department, Washington, D.C. Patton, Michael Q How to Use Qualitative Methods in Evaluation. Newbury Park, Calif.: Sage Publications. Rietbergen-McCracken, Jennifer, and Deepa Narayan, comps Participation and Social Assessment: Tools and Techniques. Washington, D.C.: World Bank Republic of Tanzania, 2000, Poverty Reduction Strategy Paper, Republic of Uganda Poverty Eradication Action Plan: A National Challenge for Uganda. Ministry of Finance, Planning and Economic Development, Kampala Monitoring and Evaluating Accountability and Transparency of Schools and Districts for UPE Funds. Ministry of Education and Sports, Kampala a. Uganda Poverty Status Report, Ministry of Finance, Planning and Economic Development. Kampala b. Five Year Strategy for Poverty Monitoring and Policy Analysis. Planning and Poverty Eradication Section, Ministry of Finance, Planning and Economic Development, Kampala Rossi, Peter H., and Howard E. Freeman Evaluation: A Systematic Approach. 2d ed. Beverly Hills, Calif.: Sage Publications. Valadez, Joseph, and Michael Bamberger, eds Monitoring and Evaluating Social Programs in Developing Countries: A Handbook for Policymakers, Managers, and Researchers. EDI Development Studies. Washington, D.C.: World Bank. van de Walle, Dominique Assessing the Poverty Impact of Rural Road Projects. World Bank, Development Research Group, Washington, D.C. Weiss, Carol H Evaluation. 2d ed. Upper Saddle River, N.J.: Prentice Hall. World Bank Building Evaluation Capacity. Lessons and Practices 8. Operations Evaluation Department, Washington, D.C. 33

29

30

31

32

33

34

35

36

37

38

39

40 FCND DP No. 111 FCND DISCUSSION PAPER NO. 111 AN OPERATIONAL TOOL FOR EVALUATING POVERTY OUTREACH OF DEVELOPMENT POLICIES AND PROJECTS Manfred Zeller, Manohar Sharma, Carla Henry, and Cécile Lapenu Food Consumption and Nutrition Division International Food Policy Research Institute 2033 K Street, N.W. Washington, D.C U.S.A. (202) Fax: (202) June 2001 FCND Discussion Papers contain preliminary material and research results, and are circulated prior to a full peer review in order to stimulate discussion and critical comment. It is expected that most Discussion Papers will eventually be published in some other form, and that their content may also be revised.

41 ii ABSTRACT Development institutions and projects frequently seek to target poorer segments of the population. Yet, existing methods for evaluating their outreach are generally unsuited to most operational settings, since they are either too costly and cumbersome (e.g., detailed income or household surveys), or they produce results that are not comparable between villages or regions within a country (e.g., participatory poverty appraisals). This paper presents a new and operationally suitable method to measure the poverty of clients of development projects in relation to the general population of nonclients. The method was developed in response to demands by donors and development practitioners for a low-cost evaluation instrument that could be used as a regular operational tool for assessing the poverty outreach of a development project or institution. While the method was originally developed for the purpose of assessing the poverty outreach of microfinance institutions (MFIs), we believe the method can be used for any development policy or project that pursues an explicit objective of reaching poorer people. The paper begins by discussing existing methods of poverty assessment. Next, the paper presents heuristic steps for identifying indicators of poverty to be tested in the case studies, including the questionnaire that was field tested in four countries with large differences in poverty-level, socioeconomic, and cultural contexts, and with MFIs that worked either in urban, rural, or mixed areas with different target clientele and financial

42 iii products. The authors then describe the method of principal component analysis used to construct a poverty score as the measure of relative poverty. The paper concludes with a summary of results from four country case studies (two in Sub-Saharan Africa, one in South Asia, and one in Central America).

43 iv CONTENTS Acknowledgments... vi 1. Introduction Choice of Methodology... 2 Design Constraints for the Development of the Poverty Assessment Tool... 2 Selection Among Alternative Methods for Assessing Poverty... 3 Method 1: Computation of a Poverty Line Based on Household Expenditures... 4 Method 2: Rapid Assessment and Participatory Appraisal... 8 Method 3: Constructing a Poverty Index Based on a Range of Indicators Multiple Dimensions of Poverty and Choice of Indicators for Case Studies Indicators in the Final Recommended Questionnaire Estimation of the Poverty Index With Principle Component Analysis Human Resources Dwelling Assets Food Security and Vulnerability Interpretation of Results Microfinance Institution A Microfinance Institution B Microfinance Institution C Microfinance Institution D Concluding Remarks Annex 1: Cumulative Distribution for Clothing Expenditures per Capita, by MFI and Client Status Annex 2: Recommended Questionnaire References... 42

44 v TABLES 1 Indicators in the final recommended questionnaire Indicators selected to represent the poverty index, by countries Relative poverty ranking of client versus nonclients...30 FIGURES 1 Indicators and underlying components Histogram of the standardized poverty index (MFI-B) Constructing poverty groups MFI-A: Distribution of client and nonclient households across poverty groups MFI-B: Distribution of client and nonclient households across poverty groups MFI-C: Distribution of client and nonclient households across poverty groups MFI-D: Distribution of client and nonclient households across poverty groups...29

45 vi ACKNOWLEDGMENTS We thank the Policy Advisory Committee of CGAP and participants of a virtual meeting, who provided useful and critical comments. We also thank the managers and staff of the four participating microfinance institutions. Brigit Helms of CGAP deserves special thanks for valuable comments and critical questions that greatly improved the tool and its presentation. A manual (Henry et al. 2000) describing each analysis step in implementing the tool can be downloaded at CGAP s website: http// All remaining errors are ours. Manfred Zeller Institute of Rural Development University of Göttingen, Germany Manohar Sharma International Food Policy Research Institute Carla Henry Consultant Cécile Lapenu Consultant

46 1 1. INTRODUCTION The reduction of poverty is an explicit or implicit objective of most development policies and projects, so that targeting of policies and project services to the poor is important in developing practice. However, the lack of simple, low-cost tools for assessing whether a project reaches the poor results in either no project monitoring or monitoring activities that use simple but crude descriptions of project beneficiaries (such as the share of women, farm size, or occupation of program beneficiaries), or in rapid or participatory assessments that are not well-suited for within- or between-country comparisons. This paper describes an operational tool developed over the past two years by the International Food Policy Research Institute (IFPRI) with technical and financial support of the Consultative Group to Assist the Poorest (CGAP). The tool was designed to assess the poverty level of project beneficiaries in relation to the general population in the intervention area. To be useful to policy analysts, donors, and development practitioners, the tool must meet reasonable time as well as cost constraints, i.e., the evaluation procedure must be completable in a few months, with the average cost per assessment not exceeding US$10,000. After a review of the different methods of poverty assessment that guided our choice of methodology, we describe the properties of the poverty assessment tool in detail. The tool was tested in collaboration with microfinance institutions (MFIs) in four case studies: one in Latin America, two in Sub-Saharan Africa, and one in Asia. We

47 2 believe the tool has a potential for broader application, however, not only in the field of microfinance. The methodology is applicable for other development interventions, with clearly defined target groups such as various types of income transfer and public work programs as well as other food and social security-related interventions. However, it is recommended that future research studies validate the tool by comparison with established national poverty benchmarks CHOICE OF METHODOLOGY DESIGN CONSTRAINTS FOR THE DEVELOPMENT OF THE POVERTY ASSESSMENT TOOL At the outset of the research project, a number of design parameters for the development of the poverty assessment tool were defined: The tool should be implementable by national research organizations and consultants in developing countries that have prior experience with socioeconomic surveys and statistical analysis; 1 The tool was developed by IFPRI with technical and financial support of CGAP. Among the different development interventions, microfinance is increasingly viewed as a way to enable the poor to carry out profitable self-employment activities. The need to reach out to the poor through microcredit was reemphasized at the Micro-Credit Summit in 1997, but many practitioners, donors, and researchers perceive a trade-off between financial sustainability and depth of outreach, although the exact nature of the trade-off is not well understood (see, for example, Hulme and Mosley 1996). At present, no operational, low-cost tool exists for measuring the depth of poverty outreach of MFIs in a way that allows for within- or between-country comparisons. The lack of a practical poverty assessment tool was the motivation for this collaboration.

48 3 The cost of implementing the tool should be relatively low (averaging US$10,000); The time frame for assessment should be short, i.e., the final results available within a couple of months; The tool should measure the poverty level of microfinance clients in relation to the general population in the operational area of the MFI; The tool should be flexible and general enough to be suitable for use in both urban and rural areas in developing countries; Results should be readily interpretable and comparable across programs within countries and, if possible, also between countries. SELECTION AMONG ALTERNATIVE METHODS FOR ASSESSING POVERTY The characteristics of poverty are multidimensional, encompassing various aspects of a household s economic and social status. Capturing these dimensions requires both qualitative and quantitative indicators. In development practice, three major types of poverty assessment methods are generally used: Construction of a poverty line and computation of various poverty measures that take into account the way in which actual household expenditures fall short of the poverty line (Ravallion 1994; Foster, Greer, and Thorbecke 1984; Moser et al. 1994, Streeten 1994);

49 4 Rapid appraisal and participatory appraisal methods in which households are ranked with respect to their wealth by community members themselves (Bilsborrow 1994); Construction of a poverty index using a range of qualitative and quantitative indicators (Hatch and Frederick 1998; Chung et al. 1997). The discussion below of the advantages and disadvantages of the three methods lays the basis for our choice of a tool based on the third method. Method 1: Computation of a Poverty Line Based on Household Expenditures The standard practice in poverty analysis has been to use household total expenditure as the primary measure to evaluate the standard of living of households (Grootaert 1983, 1986). Nationally representative household surveys such as the Living Standard Measurement Survey conducted by the World Bank are typically used to estimate poverty line and measure incidence of poverty. The criteria used in assessing whether a household is poor is based on an evaluation of whether household income is sufficient to meet the food and other basic needs of all household members. To make the assessment, a basket of goods and services corresponding with local consumption patterns and satisfying a pre-set level of basic needs for one person is constructed and valued at local consumer prices to compute its minimum cost. The value of this basket is called the poverty line, and is most commonly expressed in per-capita terms. If the per-capita income of household members

50 5 is below the poverty line, the household and its members are considered poor. If this does not hold, the household is categorized as nonpoor (Aho, Larivière, and Martin 1998; Lipton and Ravallion 1995). The advantage of Method 1 is that it is a widely accepted measure of poverty as far as its economic dimension is concerned. However, the data requirements of this method are very steep, and very comprehensive questionnaires are needed to collect it. The standard practice is to record food expenditures, using a recall period of one week and a combination of monthly or yearly recall periods to collect information on various nonfood expenditures. Even though poor households in developing countries consume a small number of goods, given the long recall periods, accuracy in reporting is a valid concern. Second, even if consumption items can be accurately recalled, ways have to be found to value home-produced foods when market prices are lacking; irregular weights and measures cause problems in computation of quantities; and information on a number of high-value items (e.g., rental value of housing) is likely to be seriously deficient. Of course, the scale of these problems can be substantially minimized through extensive training of interviewers, multiple household visits, and cataloging of informal weights and measures. However, the effect on the survey cost and the time required to address these problems are likely to be prohibitive. Moreover, the analysis of expenditure data necessitates advanced skills in statistical data analysis, which translates into high costs for data analysis as well. The costs of the survey could potentially be reduced if the evaluator had access to data on a national benchmark poverty indicator established in a previous national

51 6 household survey on poverty. If these data are accessible, one could choose to undertake a similar household survey only for MFI clients and to compare these results with the national poverty benchmark for the general population. This kind of comparison has been recently done by Navajas et al. (2000) in Bolivia. One of the major strengths of this approach is that a nationally accepted poverty measure is used and the issue of how to measure poverty is avoided. In the case of Bolivia, the national poverty benchmark is the so-called Index of Fulfillment of Basic Needs (IFBN), similar to the Method 3 discussed below. This index comprises about 10 indicators capturing housing quality, access to public services, education, and access to informal and formal health services; the indicators are combined using weights that have been determined by a form of consultative process among national poverty experts and policy analysts. Navajas et al. (2000) obtained information on these indicators in their survey of MFI clients (with the exception of access to health services) and computed the index so as to be comparable to the already-available national benchmark. This is a useful approach especially in the case where a relatively simple poverty index has been established at the national level. However, the approach of comparing information on the MFI clients with a national poverty benchmark for the general population in the country is not universally applicable for the following reasons. First, the method is only applicable in developing countries that have already undertaken a national poverty study on the basis of which a poverty benchmark index or a poverty line has been established. Second, even when countries have an established poverty measure, the measure is usually based on expenditure data rather than on a range of qualitative indicators (as indeed is the case in

52 7 Bolivia). When this is so, a very detailed and time-intensive expenditure survey of MFI clients is required, usually violating the design parameters. Moreover, whenever a substantial time lapse occurs between the national survey and the assessment exercise, factors such as inflation and changes in relative prices are likely to make comparisons difficult or even plainly inadvisable. Usually, tackling such issues requires advanced analytical skills and access to the source data from the national survey. Third, in many countries, there is likelihood that access to the national data may be restricted or the government is reluctant to release it. Other countries (e.g., China) sell the data at costs that exceed the field research budget envisaged for this tool. In still other countries, the data may be poorly documented so that considerable time is needed for a skilled poverty analyst to make the data comparable and resolve the issues of inflation and changes in relative prices. Fifth, to assure valid comparison, data collection methodology used in the program assessment exercise must closely replicate the method used in the national poverty assessment. This requires a level of collaboration that may either be unfeasible or too expensive. Finally, as MFIs are expected to operate in above-average regions or cities of a country, an assessment should be able to account for selection bias in program placement (Sharma and Zeller 1999). In summary, while comparison with existing national benchmarks may provide valid assessment of poverty outreach of a MFI, it is, in practice, often unfeasible. We conclude that the tool, to be universally applicable, must collect poverty data for nonclients to assess the relative poverty of MFI clients.

53 8 Method 2: Rapid Assessment and Participatory Appraisal Rapid Appraisal (RA) and Participatory Appraisal (PA) are grouped together as the second method. The two approaches are often thought to be the same, since they seek input from community members using similar techniques, e.g., wealth ranking and community mapping. There are differences, however (Bergeron, Morris, and Medina Banegas 1998). The ultimate goal of PA is empowerment of the target group. This necessitates extensive participation by the community and assumes an open research and development agenda. This can hardly be done within one or two days. RA methods, on the other hand, are meant to provide evaluators data on the community in a very short time. RA requires the participation of the community, but the timeframe is usually a oneday visit to the community and the agenda of the inquiry is predetermined. RA and PA methods are widely used and accepted tools for identifying vulnerable groups in a community (Bilsborrow 1994; Boltvinik 1994; Hatch and Frederick 1998). They are extensively used by development programs and institutions, including MFIs, for targeting services to poorer clients. The RA method, in particular, has relatively low time requirements for data collection. While these methods can be well suited for targeting and for the participatory design of development projects and services, a number of disadvantages exist for assessing poverty for purposes of regional, national, or international comparisons (see, also, Chung et al. 1997). First, the results are difficult to verify, because they stem from the subjective ratings of community members. Thus, the results are difficult to compare across geographic locations or programs in a country (Chung et al. 1997). Second, the approach is likely to find poor people in any community,

54 9 and the percentages of poor people may not vary much across villages. In other words, the method may be consistent in finding the poorest third in one village, but it may not be consistent in finding in which communities the poorest third of an entire region reside. Third, as the results are hard to verify a problem with household expenditure as well, strategic responses that make everybody or certain groups of the community poorer cannot be ruled out as the respondents may expect to receive benefits, such as access to financial services, after the completion of the poverty assessment. To avoid this kind of bias or strategic responses, verifiable indicators should be used as much as possible. Finally, the PA method requires skillful and experienced communicators who will command higher salaries than enumerators who are required only to apply a structured and formalized questionnaire. For national and international comparisons, there could be concern about the bias introduced by the way that PA is implemented. Thus, while we agree with Chung et al. (1997) that these methods are useful and operational for targeting services by specific development programs, including MFIs, they considerably violate the design constraints spelled out above. Method 3: Constructing a Poverty Index Based on a Range of Indicators A third method of assessment is to identify a range of indicators that describes different dimensions of poverty and for which credible information can be quickly and inexpensively obtained. Once information on the range of indicators has been collected, the indicators may be aggregated into a single index of poverty by using some weighting scheme. One well-known application of this method is the Human Development Index

55 10 (HDI) (UNDP 1999; Annand and Sen 1994), which is based on three components: educational attainment, life expectancy at birth, and per-capita income adjusted for purchasing-power parity. The national poverty index for Bolivia, mentioned above, also follows this method. A particularly popular example of the indicator method is the housing index used by many MFIs (in particular, in South and Southeast Asia) for targeting financial services to poorer clients (Hatch and Frederick 1998). Among its major advantages is that the list of indicators contributing to the housing index, such as quality of roof or walls of the house, can be obtained very quickly through inspection. Also for this reason, strategic misrepresentation in responses can be minimized. Another strength of the housing index is that the housing indicators can be easily adjusted to local peculiarities. For example, different types of roofs used for the housing can be distinguished in consideration of locally available roofing materials in the area. Because of its operational ease and the low costs of obtaining and analyzing information, the housing index is widely used by MFI practitioners, particularly in Asia, where the concept originated. At present, its major use is to identify poor households for targeting by program services. Among the disadvantages is that the weights assigned to individual indicators in the aggregation process are somewhat arbitrarily set by the development institution that applies the index. In fact, because of differing housing conditions across (or even within) countries, proponents of this approach call for adjusting the weights from country to country as well as including alternative indicators of poverty (see Hatch and Frederick 1998; Gibbons and DeWit 1998). However, precisely because of the arbitrarily set weights, comparisons

56 11 within and across countries can be questionable. Another major disadvantage of the housing index is that the index only focuses on a single dimension of poverty (e.g., housing), neglecting other important dimensions such as food security, vulnerability, and human capital. In principle, the time and cost requirements of the indicator method in terms of data collection and analysis can be relatively low. It can be valid if several dimensions of poverty are included. For these reasons, the indicator method was chosen as the basis for the poverty assessment tool. The tool seeks to build on the many strengths of the housing index, but allows for a rigorous and standardized procedure for determining the weights and including alternative, location-specific indicators. Moreover, the measurement scale has been standardized to allow comparisons of programs between countries. The choice of the indicator method led to the following working steps in developing the tool: 1. Identifying a large number of indicators that reflect poverty levels powerfully and for which credible information can be quickly and inexpensively obtained; 2. Designing a survey methodology that facilitated the collection of information on these indicators from households living in the operational area of the MFI; and 3. Applying a suitable statistical methodology for summarizing information contained in the various indicators into a single summary index that could be used to make poverty comparisons between households.

57 12 The initial compilation of indicators for this approach was based on a detailed review of results of large, in-depth surveys on household economics as well as of indicators and methods used by MFIs, famine early warning systems, and national monitoring systems for food security, nutrition, and vulnerability (see, for example, Wratten 1995; Radimer et al. 1992). 3. MULTIPLE DIMENSIONS OF POVERTY AND CHOICE OF INDICATORS FOR CASE STUDIES Because of the multifaceted nature of poverty, we do not recommend reliance on any one dimension such as housing, food security, or access to education. Rather, to capture different dimensions of poverty, we used two groups of indicators to develop a generic questionnaire that was then field-tested with four MFIs, one in Latin America, two in Sub-Saharan Africa, and one in Asia. The first group of indicators expresses the means to achieve welfare. These reflect the income potential of households and their members and relate to the household s human capital (family size, education, occupation, etc.), physical capital (type and value of assets owned), and social capital (for indicators on social capital, see, for example, Grootaert 1998). The second group includes indicators related to achievements in consumption in order to fulfill present and future basic needs (namely access to health services, food, electricity, energy, water, shelter and clothing, human security, and environmental quality). Studies comparing different indicators based on income and

58 13 consumption conclude that recommending one measure over another is difficult (Skoufias, Davis, and Soto 2000). However, consumption over time (seasons or years) is more stable than income, and households provide information more easily on what they consume than on what they earn. For this reason, our method heavily relies on consumption indicators, although the first group of indicators expressing means available to the household to increase its standard of living is also included. In coming up with reliable indicators, the key challenge is to identify key components of consumption that are either unambiguous measures of poverty (such as incidence of hunger) or those that correlate well with or are good proxies for total household expenditures. Hence, it is not necessary to compile all the food and nonfood expenditures of a household, since some types of expenses are closely related to the level of poverty of a household, and others not. For example, studies have shown that the proportion of clothing expenditures in the household budget remains stable, around 5 10 percent of total expenses (Aho, Larivière, and Martin 1998; Minten and Zeller 2000). A recent study by Morris et al. (1999) found clothing expenditure to be one of the expenditure components that increased proportionally with total household expenditures. Since clothing, unlike food commodities, usually requires the purchase of either the finished garment or materials to make a garment, it also avoids the valuation problem associated with imputing costs for home-produced goods. The preselection of over 300 indicators belonging to the above two principal groups that were tested in the four case studies was based on the following steps:

59 14 1. Extensive literature review and expert consultation on the general availability and use of poverty indicators, including indicators developed for operational project monitoring and poverty targeting (see Hatch and Frederick 1998; Chung et al. 1997); 2. Use of eight-point criteria to evaluate indicator suitability, namely: suitability for rural and urban contexts, sensitivity of question, time and cost requirements to obtain answer, quality of the indicator in discriminating between different poverty levels, reliability (including the possibility to verify the answer in a recheck), simplicity, and universality in an international context; 3. Development of a generic questionnaire for testing selected indicators in four case studies; 4. Adaptation of the questionnaire in the field to account for local-level specificities; 5. Obtaining information for each indicator through a structured household survey. It is, of course, not surprising that the preselected indicators contained many that are used by national poverty and vulnerability monitoring systems as well as development programs, including MFIs. For example, all of the indicators used to construct the housing index and indicators of the net worth test used by the Grameen Bank were included. The questionnaire was field tested in four countries with large differences in poverty-level, socioeconomic, and cultural contexts, and with MFIs that worked either in urban, rural, or mixed areas with different target clientele and financial products. The

60 15 selection of the case study countries was guided by the desire to have as much heterogeneity as possible as well as to conduct field testing with the aim of reducing the number of indicators to be included in the final recommended questionnaire. This was done through the following steps: 1. identifying indicators independently for each case study that are tightly related to poverty levels; 2. identifying indicators that are suitable to all four countries (i.e., those that are robust to diverse socioeconomic and cultural contexts); 3. identifying indicators suitable for capturing local specificities and evaluating their importance in the overall assessment; 4. cataloging cost and other constraints and problems of the survey tool and related analysis encountered in the case studies; 5. sharing the results with the selected MFIs and other stakeholders so as to critically evaluate the method; 6. developing, testing, and standardizing a method that sums up the different indicators in one poverty index and allows comparisons of relative poverty outreach between MFIs and countries; and 7. documenting all procedures involved in a user-friendly manual to support future independent assessments.

61 16 4. INDICATORS IN THE FINAL RECOMMENDED QUESTIONNAIRE Steps 1 to 6 above led to the selection of the final list of indicators (Table 1) for the recommended questionnaire (Annex 1). The selection of this list was based on a number of criteria, including the ease and accuracy with which information on the indicators could be elicited in the household survey and the significance of the correlation of the indicator with per-capita expenditure on clothing and footwear (the poverty benchmark indicator). 2 The indicators that were selected for the final recommended questionnaire are listed in Table 1 (Henry et al. 2000) (see Annex 2 for the full version of the recommended questionnaire). Table 1: Indicators in the final recommended questionnaire Human resources Dwelling Food security and vulnerability Assets Others Age and sex of Number of rooms Number of meals served in last Area and value Nonclient s adult household Type of roofing two days of land owned assessment members Type of exterior Serving frequency (weekly) of Number and of poverty Level of walls three luxury foods value of outreach of education of Type of flooring Serving frequency (weekly) of selected MFI adult household Observed one inferior food livestock members structural Hunger episodes in last one resources Occupation of condition of month Value of adult household dwelling Hunger episodes in last 12 transportationrelated assets members Type of electric months Number of connection Frequency of purchase of staple Value of children below Type of cooking goods electric 15 years of age appliances fuel used Size of stock of local staple in in household Source of dwelling Annual clothing/ drinking water footwear expenditure for Type of latrine all household members 2 Per-capita clothing expenditure was chosen as the benchmark indicator since it bears a stable and highly linear relationship with total consumption expenditure, a comprehensive and widely accepted measure of poverty. Annex 1 shows the cumulative distributions for this benchmark indicator for the four countries.

62 17 5. ESTIMATION OF THE POVERTY INDEX WITH PRINCIPLE COMPONENT ANALYSIS The use of multiple indicators enables a more complete description of poverty, but it also complicates the task of drawing comparisons. The wide arrays of indicators have to be summarized in a logical way, underlining the importance of combining information from the different indicators into a single index. The creation of an index requires undertaking the difficult task of finding a set of weights that can be meaningfully applied to different indicators so as to come to an overall conclusion. The usual practice is for the evaluator to set the weights himself, taking account of local conditions but otherwise involving a significant degree of arbitrariness. However, in order to allow comparisons across MFIs within and across countries, an aggregation method was required that would evaluate each indicator and determine the weights in a standardized and rigorous way. We expected, however, that the relative strengths of different indicators in predicting poverty were very likely to vary across regions and countries. This explained our preference for a method that allows adjusting weights for each situation, taking into account the specific poverty context existing therein. For example, for the case of nutritional indicators, Habicht and Pelletier (1990) show that context matters in the choice of appropriate nutrition-related indicators. Moreover, the aggregation method should allow the testing and eventual inclusion of indicators that are location-specific and are recommended by national experts. Thus, while the indicators in Table 1 are recommended, the poverty assessment tool allows the inclusion and testing of

63 18 additional local indicators. For example, in Nicaragua, a large share of rural households had members who worked abroad and improved the living standard of the family through remittances. Hence, a remittance-related indicator was deemed important. The method of principal component (PC) analysis, when used as an aggregation procedure, addresses most of the concerns raised above in an objective and rigorous way (see, for example, Temple and Johnson [1998] and Filmer and Pritchett [1998, 1999]). 3 Specifically, PC analysis isolates and measures the poverty component embedded in the various poverty indicators and creates a household-specific poverty score or index. Relative poverty comparisons are then made between client and nonclient households based on this index. Basically, the principal component technique slices information contained in the set of indicators into several components. Each component is constructed as a unique index based on the values of all the indicators. The main idea is to formulate a new variable, X *, which is the linear combination of the original indicators such that it accounts for the maximum of the total variance in the original indicators. That is, X * is computed as X = w1 X 1+ w2 X 2 + w3 X 3 *, 3 Because of lack of income and expenditure data, Filmer and Pritchett (1998) and Sahn and Stifel (2000) use the principal component method, and apply it to national household data for India and for data from the Demographic and Health Surveys of various African countries, respectively. Filmer and Pritchett (1998) estimate the relationship between household wealth and the probability that a child is enrolled in school. As a proxy for household wealth, they constructed a linear asset index from a set of asset indicators, using a principal component technique. They conclude that this index is robust, produces internally coherent results, and provides a close correspondence with available economic data at higher aggregation levels. Filmer and Pritchett (1998) then validate this method with other datasets from Nepal, Indonesia, and Pakistan, which contain asset indicators and consumption expenditures as well. They find that the asset index has reasonable coherence with current consumption expenditures and works as well or better than traditional expenditure measures in predicting enrollment status.

64 19 where the weights (the ws) are specified such that X * accounts for the maximum variances in X 1, X 2, and X 3. This index has a zero mean and a standard deviation equal to one (Basilevsky 1994; Sharma 1996). The PC analysis therefore extracts underlying components from a set of information provided by summary indicators. In the case of this poverty assessment tool, information collected from the questionnaires make up the indicators, and the underlying component that is isolated and measured is poverty. The first principal component accounts for the largest proportion of the total variability in the set of indicators used. The second component accounts for the next largest amount of variability not accounted by the first component, and so on for the higher order components. In the example presented in Figure 1, PC analysis uses the information on the comovement among the indicators to isolate and quantify the underlying common components, e.g., poverty and demography. The poverty component is expected to account for most of the movements in the indicators and will be the strongest of all the components. The poverty component can be easily identified by analyzing the signs and size of the indicators in relation to the new component variable. For example, according to theory, education level should contribute positively not negatively to wealth. PC analysis, hence, can be used to compute a series of weights that mark each indicator s relative contribution to the overall poverty component. Using these weights, a household-specific poverty index (or score) can be computed based on each household s indicator values.

65 20 Figure 1: Indicators and underlying components Components Poverty Demographic characteristics Indicators Human resource indicators Dwelling indicators Asset indicators Food indicators Other indicators In our case, the indicators contributing to the index were selected in two stages of statistical analysis. First, the strength and significance of the correlation of each of the initial 300 indicators used in the test studies with the poverty benchmark indicator, i.e., per-capita clothing expenditure, was tested. Only those that are significantly correlated (with a probability of error less than 10 percent) with the benchmark indicator were submitted to the second stage involving principal component analysis. In each of the four case-study countries, indicators passed the first stage. In the second stage, various criteria and cutoff values concerning the results 4 of the PC analysis were used across all 4 The following criteria or levels for accepting an indicator recommended in the manual are (1) the component loading has the expected sign following theory; (2) the values for communality for an indicator should be above 0.2, and the overall PC model should have a Kaiser-Meyer-Olkin (KMO) index above 0.6. These criteria were followed in all four case studies, and led after having them prescreened by their correlation with the poverty benchmark to the selection of about 20 indicators for contributing to the country-specific poverty index.

66 21 case studies to accept or reject indicators. Table 2 contains the list of indicators for the four country studies that were selected by the standard two-stage selection procedure. 5 Each of the four case studies uses indicators that combine different dimensions of poverty concerning human resources, housing conditions, assets, and food security and vulnerability. It is noteworthy that nine indicators (out of a potential 300) were commonly used in three of the four cases studies. HUMAN RESOURCES Eight indicators related to human resources were used in the four case studies. These indicators reflect the level of education in the household and the presence of unskilled labor force. The percentage of wage laborers in the household seems to be particularly important in the relatively poorer countries of Southern Africa and South Asia (MFI-C and MFI-D). The indicator expressing the level of education of the household head was used in three out of four countries. DWELLING Dwelling indicators discriminated between relative poverty levels well. In the case of MFI-D in South Asia, 8 of 20 indicators were related to housing quality. The importance of dwelling indicators in South Asia supports the use of the housing index as 5 Cumulative frequency distribution of per-capita clothing and footwear expenditure by client and nonclient households is provided for each of the case studies in Annex 1.

67 22 Table 2: Indicators selected to represent the poverty index, by countries Poverty indicator MFI-A MFI-B MFI-C MFI-D # Human resources Maximum level of education in household x x 2 2. Percent of adults who are wage laborers x x 2 3. Education level of household head x x 2 4. Percent of literate adults in household x 1 Dwelling Value of dwelling x x 2 2. Roof made of permanent material x x 2 3. Walls made of permanent material x x 2 4. Quality of flooring material x 1 5. Electric connection x x x 3 6. Source of cooking fuel x x 2 7. Latrines in the house x x x x 4 8. Number of rooms per person x x x 3 9. Access to water x x Structure of the house x 1 Assets Irrigated land owned x 1 2. Number of TVs x x 2 3. Number of radios x 1 4. Number of fans x x 2 5. Number of VCRs x 1 6. Value of radio x 1 7. Value of electrical devices x x x 3 8. Value of vehicles x 1 9. Value of assets per person/adult x x x 3 Food security and vulnerability Number of meals served in last two days x 1 2. Episodes of hunger during last 30 days x x x 3 3. Episodes of hunger in last 12 months x x x x 4 4. Number of days with luxury food 1 x x x 3 5. Number of days with luxury food 2 x x x 3 6. Number of days with inferior food x x 2 7. Frequency of purchase of basic good x x 2 8. Frequency of purchase of basic good x 1 9. Food stock in house x Use of cooking oil x 1 Miscellaneous indicator Per person expenditure on clothing x x 2 Total number of indicators

68 23 an important indicator of poverty in that region. However, in the African cases (MFI-B and MFI-C), where housing is relatively homogenous, only four and five housing indicators were used, respectively. The quality of latrines appeared in all the case studies. House size (rooms per person) was used in three countries. ASSETS A total of 15 indicators on the number or value of assets is included in the four case studies. They were particularly important (5 out of 16 indicators) in the Central American country (MFI-A), the most well-off country in the sample. The amount of land possessed is important only for MFIs serving rural and agricultural areas, as is the case in MFI-D. FOOD SECURITY AND VULNERABILITY These indicators turned out to be very important in explaining differences in relative poverty in all four studies, particularly in the Southern African country (MFI-C), which is the poorest. The indicator of chronic hunger (episodes of hunger in the last 12 months) appears in all four cases. Indicators of short-term hunger (episodes of hunger in the last 30 days) and frequency of luxury food consumption during the week appeared in three cases.

69 24 6. INTERPRETATION OF RESULTS As indicated above, principal component analysis produces a household-level poverty index. Figure 2 gives an example of the distribution of the poverty index across households using MFI-B data. Figure 2: Histogram of the standardized poverty index (MFI-B) In each case study, a random sample of 300 nonclient households and 200 client households was chosen. To use the poverty index for making comparisons, the nonclient sample is first sorted in an ascending order according to its index score. Once sorted, nonclient households were divided into terciles based on their index score: the top third of the nonclient households were grouped in the higher group, the middle third in the middle group, and the bottom third in the lowest group (Figure 3). Since there are