Monitoring. Urmy Shukla J-PAL/CLEAR South Asia at IFMR. Monitoring & Evaluation Training Course for the Indian Economic Service August 1, 2013

Transcription

1 Monitoring Urmy Shukla J-PAL/CLEAR South Asia at IFMR Monitoring & Evaluation Training Course for the Indian Economic Service August 1, 2013

2 Why do some economic development projects in India fail? A. Corruption B. Lack of Money C. The Rainy Season D. Lack of Maintenance/Follow-up 25% 25% 25% 25% Corruption Lack of Money The Rainy Season Lack of Maintenance/Fol...

3 Session Outline What is Monitoring? Why Monitor? How to Monitor? How to Learn from Monitoring? J-PAL s Role in Monitoring

4 WHAT IS MONITORING?

5 The Basics of Monitoring Evaluation Program Evaluation Impact Evaluation Monitoring

6 What is Monitoring? Monitoring is the routine collection and analysis of information to track progress against set plans and check compliance to established standards Helps identify trends & patterns, adapts strategies, and inform decisions

7 Difference between is Monitoring and Evaluation? What is it? Monitoring Ongoing data collection/analysis. Track changes in context, progress toward results. Evaluation Systematic review of what has happened and why. Determine relevance, efficiency, effectiveness, impact. Why do it? Inform day-to-day decisionmaking Accountability and reporting Accountability and learning Improve current and future programs Who does it? Internal staff External consultants and program staff

8 Monitoring & Evaluation Program monitoring goes hand in hand with evaluation When evaluations reveal no impact, it is often the result of faulty implementation Without adequate monitoring (or process evaluation) it is impossible if estimate if the program treatments are effective Process Evaluation: From Inputs to Outcomes Program Evaluation: Implementation, Maintenance

9 For what type of project is monitoring most important? A. Government B. Private Sector C. NGO 33% 33% 33% Government Private Sector NGO

10 WHY MONITOR?

11 Lant Pritchett (2009) India as Flailing State Government of India s personnel are highly trained at top levels and highly capable Civil service exams amongst the most competitive in the world Excellent & well designed polices are often formulated BUT: Below the powerful administrative head the capability of the India state to implement programs and polices is weak Problems of absenteeism, indifference, incompetence, and corruption

12 Need for monitoring & accountability: Short-Term 1 in 5 government primary-school teachers and more than 1/3 of health workers absent from their facilities in developing countries (Missing in Action World Bank Paper) The fiscal cost of teacher absence in India is over $1.5 billion per year, accounting for 60 percent of the funds raised from India s special education tax in the year of the survey (2010 The Fiscal Cost of Weak Governance Muralidharan et al). Considerable waste of resources and missing opportunities to educate children and improve the health of populations.

13 Need for monitoring & accountability: Long-Term Reduced education and health levels reduce long-run growth, since public human capital investment accounts for a large fraction of total investment in many countries Corruption and weak institutions reduce private investment and, and therefore growth.

14 HOW TO MONITOR?

15 What is the best type of monitoring system? A. Top-Down B. Bottom-Up C. Need Both D. Depends 25% 25% 25% 25% Top-Down Bottom-Up Need Both Depends

16 Monitoring Structures External: Top down monitoring structures Bottom up monitoring structures - citizen social accountability user feedback monitoring Internal: Program monitoring as a tool for program management Monitoring as an incentive (program in itself)

17 Bottom-up: Citizen s Report Card-Bangalore (Paul) Citizens give a grade to service providers (hospitals, public agencies, etc.) Grades lead to inter-agency comparisons, which are made public

18 Bangalore-implemented CRC Results Feedback from the sample of low income households showed that over 70 per cent of the households had visits to the agencies 3 or more times to solve their problems. Nearly a third had to pay bribes. The report card findings were widely publicized through the press in Bangalore Newspapers created public awareness about the findings The Times of India, published the findings about each of the agencies every week Over a period of 9 years from 3 CRCs were held(1994, 1999, 2003) overall satisfaction with services increased in every sector

19 Bottom-Up: Policy Implications Internal monitoring mechanisms of government not always able to track the outcomes of its policies. Some aspects of outcomes are best known to the users and beneficiaries of programs and services

20 Top-Down Monitoring: Teacher Absenteeism Teacher absence is now widely used as a governance indicator in education In spite of substantial increases in spending on education inputs, improvements on this key measure of governance have been low

21 Top-Down Monitoring: Teacher Absenteeism The central and state governments in India have considerably increased spending on primary education over the past decade. Majority of the education budget is allocated to improving traditional input-based measures of school quality, which do not matter as much for learning outcomes as much as teacher attendance and effort. Teacher absences significantly hurt student learning, (school infrastructure, teacher qualifications, training certification not correlated with improved student learning)

22 Top-Down Monitoring: Teacher Absenteeism 23.7 percent of teachers in public schools across rural India are absent during unannounced visits to school Villages with inspections had a reduction in average teacher absence of 8.2 percent.

23 Reference material For more information please refer to: Is India a Flailing State?: Detours on the Four Lane Highway to Modernization (Lant Pritchett) Holding the State to Account, Lessons of Bangalore s Citizen Report Cards (Samuel Paul) Missing in Action: Teacher and Health Worker Absence in Developing Countries (Journal of Economic Perspectives Nazmul Chaudhury et all) The Fiscal Cost of Weak Governance: Evidence from Primary Education in India (Karthik Muralidharan et all)

24 HOW TO MONITOR: STEPS

25 The essentials of Monitoring Answers the question: What are we doing?

26 Methods of Monitoring First hand information Citizens reporting Surveys Formal reports by project/programme staff Project status report Project schedule chart Project financial status report Informal Reports Graphic presentations

27 Monitoring: Questions Is the intervention implemented as designed? Does the program perform? Plans and targets Inputs Outputs Outcomes Implementation Is intervention money, staff and other inputs available and put to use as planned? Are inputs used effectively? Are the services being delivered as planned? Is the intervention reaching the right population and target numbers? Is the target population satisfied with services? Are they utilizing the services?

28 Implementing monitoring Develop a monitoring plan How should implementation be carried out? What is going to be changed? Are the staff s incentives aligned with project? Can they be incentivized to follow the implementation protocol? How will you train staff? How will they interact with beneficiaries or other stakeholders? What supplies or tools can you give your staff to make following the implementation design easier? What can you do to monitor? (Field visits, tracking forms, administrative data, etc.) Intensity of monitoring (frequency, resources required, )?

29 HOW TO LEARN FROM MONITORING?

30 What should we do with monitoring reports? A. Give them to our superiors B. Publish them for the public C. Read and summarize our findings D. Use them when designing new projects 25% 25% 25% 25% Give them to our superiors Publish them for the public Read and summarize our... Use them when designin...

31 Admitting Failure We work in imperfect policy environments. Learn from mistakes. Don t ignore them.

32 When monitoring a cash transfer programme, we find high rates of corruption among the staff. What do we do? A. Stop the programme B. Stop the practice of cash transfers all together C. Take disciplinary action and hire new staff D. Research ways to stop the corruption 25% 25% 25% 25% Stop the programme Stop the practice of cash... Take disciplinary action a.. Research ways to stop th...

33 J-PAL S ROLE IN MONITORING

34 J-PAL use of monitoring Process Monitoring: We need to know if interventions are actually working or otherwise our test of impact is meaningless AND: We look at monitoring as interventions