Monitoring and Evaluation: A Logical Frame. Evaluating the Impact of Projects and Programs Beijing, China, April 10-14, 2006 Shahid Khandker, WBI

Transcription

1 Monitoring and Evaluation: A Logical Frame Evaluating the Impact of Projects and Programs Beijing, China, April 10-14, 2006 Shahid Khandker, WBI 1

2 What is M&E? Monitoring and evaluation are tools that make it possible to identify and measure the results of projects, programs or policies 2

3 Why the Concern? To evaluate and adjust strategies and activities To report on progress to interested parties, clients, taxpayers and the general public To identify and share with others best practices and lessons learned To improve the programming of new interventions and strategies 3

4 Monitoring Provides regular information on how things are working Defn: A continuing function that uses: A) systematic data collection and analysis of specific indicators of progress. B) Provides management with indication of extent of progress towards goals - achievement of deliverables -use of resources. C) Contributes to performance improvement. D) Conducted by business unit. 4

5 Evaluation Evaluation can only be done after a certain time and requires more thorough investigations Conducted by independent evaluators. Defn: A systematic and objective measurement of the results achieved by a project/program/policy in order to assess its relevance, the efficiency of implementation, its effectiveness, impact and sustainability. 5

6 Monitoring vs Evaluation Monitoring assesses progress in implementation of ongoing programs Evaluation provides a snapshot against some benchmarks at a point in time of programs that may or may not be continuing Monitoring looks at progress relative to targets and assumes there is causality Evaluation seeks to prove causality 6

7 Monitoring and Evaluation - Rationale Monitoring Holds implementers accountable for delivery of inputs Provides basis for corrective action Provides assessment of continued relevance Evaluation Accountability - was money well spent? Learning - what could we do better next time? 7

8 Monitoring and Evaluation, and the Chain of Results to undertake activities... Resources are mobilized the direct results of which... must have effects... This chain is based on a series of logical relationships (if then) called the logframe and an impact on development Monitoring Evaluation 8

9 Logframe of a Project or Program Resources are mobilised to undertake activities... the direct results of which... must have effects and an impact on development budget funds building, training, delivery amount of services provided, processes completed Health, literacy consumption, life expectancy, poverty Allocation Inputs Outputs Outcomes Impact 9

10 What Type of Indicators Are Needed? -MDGs- 1. Eradicate extreme poverty and hunger 2. Achieve universal primary education 3. Promote gender equality and empower women 4. Reduce child mortality 5. Improve maternal health 6. Combat HIV/AIDS, malaria and other diseases 7. Ensure environmental sustainability 8. Develop a Global Partnership for Development 10

11 What Type of Indicators Are Needed? GOAL:Improve the health status of the population through improved water supply Impact Impact on living standards Outcomes 5-10 yrs 1-3 yrs Lower poverty; prevalence of malnutrition Access, usage, and satisfaction of beneficiaries - % hh with access to safe water supply; % hh with continuous water supply during the day Outputs Goods and services generated - Newly installed or rehabilitated water points;% wastewater treated Annual Inputs Annual Resources (financial, material and human) provided - Real investment in water supply and sanitation 11

12 From project to program The new development strategy and its implications for M&E Impact Monitoring the outcome and impact of development across sectors Outcomes Outputs Inputs Focus on demand-side issues (data for decisions): logframe provides the foundation; largely project/program level: operationally-focused; tend to emphasize M&E at lower end of logframe [inputs/outputs); link to project design/implementation Reflects a greater emphasis on: countryownership, long-term capacity-building -also provides greater opportunities for civil society involvement and participatory M&E approaches. Link to PRSP 12

13 Why Monitor? To ensure best use of scarce resources To make user feedback integral to poverty reduction efforts To mitigate the standard pitfalls in service provision to the poor To enforce a concrete link between policy goals and policy inputs To incorporate the many dimensions of poverty reduction into our decisions 13

14 What to Monitor Identify few indicators, measure them well, and use the results for policy makers A prioritized list of input, output, outcome and impact indicators for monitoring Develop a data collection system to institutionalize a monitoring system: budget and administrative data, facility and other survey data. 14

15 How to Monitor Manage a monitoring system that integrates a variety of different types of information: MIS, surveys and censuses, and participatory exercises Draw up a monitoring matrix which identifies: -Data sources for each indicator -Frequency of measurements -The organization responsible for collecting information. Collect information in a sequence that maximizes the complementaries between different types of data. Outputs: -Annual progress report -Database for continuous monitoring -Desk studies 15

16 Features of Good Indicators Relevant; measure factors that reflect objectives e.g. Number of eggs produced vs. Poverty incidence Direct and unambiguous measure of progress e.g. % Hh expenditures on health care vs. Immunization coverage Varies across areas, groups, over time; sensitive to changes in policies, programs and institutions e.g. Life expectancy vs. Infant mortality rates Can be tracked frequently; not too costly e.g. Iron deficiency in pregnant women vs. Pre-natal care visits Consistent with the decision making/budgetary cycle 16

17 How Many Indicators Should Be Monitored? There is not a general rule but... It is more useful to identify a small number of indicators, measure them well and use the results for policymaking than to enumerate a long list of indicators, measure many of them badly and not use the results for policy decisions at all 17

18 At What Level of Disaggregation? Aggregate indicators are useful for giving an overall picture of country situation or cross-country comparisons However, they tend to mask differences across areas, gender, or social group Appropriate level of disaggregation depends on country conditions and the indicator itself 18

19 Types of Disaggregation Geographical Rural/urban Administrative units (states, districts, municipalities, etc) Geoclimatic zones Gender Income or consumption level Socially defined groups 19

20 Experience to Date Lack of consistency between goals and indicators for monitoring Multitude of indicators without clear prioritization Unbalance between intermediate (input/output) and final (outcome/impact) indicators Inadequate/unclear indicators 20

21 Purposes of an M&E People in charge Decide Manager, Operational staff Know, be informed 4main purposes Manage, monitor Public, beneficiaries, contributors Know, understand and learn lessons Researcher 21

22 Key M&E Products Start Before During At the end Afterwards Program To check the design To improve the implementation For accountability purposes To assess the impact 22

23 Components of a M&E Strategy Outcome-based monitoring system Complemented with systematic impact evaluation Creating a feedback process Building capacity for monitoring and evaluation Promoting participation 23

24 Outcome-based Monitoring System Setting goals and targets (including establishing the baseline) where do we want to go? Identifying indicators that can be used to measure progress towards goals Collecting data - what progress is being made? Providing feedback for decision making what needs to be changed along the way 24

25 Monitoring System: Where to Start? Stock taking: -Who are the actors? What are their activities? What is their capacity? What are their roles and needs? Review the actions currently undertaken to build or re-enforce capacity. Review of needs in information 25

26 Monitoring System: What Limitations? A series of issues are of concern: - Roles badly defined - Lack of coordination - Lack of reliability and relevance of information - Difficulty in accessing information - Long delays in production of information - Lack of use of the data by the users 26

27 Example: Monitoring Poverty Reduction 27

28 Monitoring poverty reduction requires Measuring poverty Understanding poverty reduction strategies Monitoring outcomes and impacts goes beyond project-based monitoring Evaluating m&e system and its capacity 28

29 Measuring poverty Poverty is a multi-dimensional concept income, consumption, health, education, water, housing, sanitation, capability and empowerment Poverty is also dynamic vulnerability and coping mechanism Poverty is also gender, caste, ethnic, and geographical based 29

30 Poverty reduction strategies Market matters; state matters too they can t be substitutes; a right balance is required. Involve the poor in development process poverty reduction requires both growth and targeted measures Development strategy needs (a) policies to improve the investment climate to raise income and employment growth, and (b) policies to equip and empower people to participate in growth. 30

31 Better investment climate and better participation Macroeconomic stability and openness Good governance and strong institutions Quality infrastructure Growth reduces poverty but not all growth is equally pro-poor Empowerment through social, human, and infrastructural investment to enable poor to participate in growth 31

32 Setting up an M&E system for Poverty Monitoring 1. Defining information needs. Deciding on indicators 2. What tools do we need? What tools do we have? 3. Defining institutional functions (including feedback processes) 4. Defining a 5-10 year work program 5. Defining resource requirements 32

33 Step 2 Measuring outcomes What tools do we need? What tools do we have? 33

34 Measuring outcomes - What tools do we need? What tools do we have? Tools for measuring the extent and depth of poverty (poverty analysis) Tools for monitoring changes over time (poverty dynamics) Tools for comparing differences over space (poverty mapping) Participatory poverty monitoring (listening to the poor) 34

35 Measuring poverty and data quality Tools: - A good household survey (there are various options - HBS, LSMS etc.) Measuring poverty is a data hungry exercise (means and distributions) Household Consumption (If you re going to measure it - do it properly) Sampling and non-sampling error Monitoring over time compounds the error problem 35

36 Monitoring changes over time Tools: Administrative data/mis; Institutionbased surveys; household surveys Monitoring leading indicators (Service Delivery) Key attributes of an indicators survey Simple to execute Rapid reporting Disaggregatable to low levels Core Welfare Indicators Questionnaire (CWIQ) 36

37 Monitoring Spatial/regional differences Main source of information on distributional outcomes - household surveys - permit only limited disaggregation. Very large data sources (e.g. census) typically collect very limited information on welfare outcomes. Impute a preferred measure of welfare (e.g. comprehensive real consumption) from household survey into census, using statistical prediction methods 37

38 What are Poverty Maps? 38

39 Participatory Poverty monitoring (listening to the poor) Limitations of statistical information Qualitative and participative approaches Complementary not alternative approaches 39

40 Data to monitor investment climate and empowerment Firm and institutional surveys Good governance indicators Empowerment indicators Community participation indicators Data on quality of infrastructure Both quantitative and quality information required 40

41 Step 3 - Setting up a 5- year work program National household budget survey with social indicators (LSMS), firm and institutional surveys Annual monitoring survey (CWIQ) Participatory poverty assessments (PPA) Population and other census Consumer and producer price collection Household panel study Timely national accounts Current public expenditure statistics 41

42 Step 4-Defining Institutional functions Illustrative Structure of Poverty and Welfare Monitoring System National Poverty reduction committee Sr. Govt. Officials Welfare Monitoring Unit Welfare Monitoring Advisory Group National statistics office Surveys, prices, natl.accounts Line Ministries Program & input monitoring Admin. financial data P.P.A Group Qualitative studies University Panel studies 42

43 Step 5 - Defining Resource requirements National budgets Coordinated donor support through national budgets Statistical capacity-building trust fund 43

44 Key messages Monitoring poverty reduction goals goes beyond selection of indicators and measuring inputs and outputs A range of different monitoring tools are required Poverty reduction monitoring goals should include statistical capacity-building as a core element Must be a national commitment to funding monitoring and statistical systems Must ensure existence of appropriate incentives for good M&E? 44 Feedback into budgeting and policy process