Identifying problems and questions

Transcription

1 Identifying problems and questions Shubhra Mittal Senior Policy and Training Manager March 29, 2017

2 I. Programme Lifecycle II. Components of Programme Evaluation Identifying problems and questions III. Conclusion

3 Lifecycle of a Programme CONCEPTUALISATI ON & DESIGN PROGRAMME IMPLEMENTATION PROGRAMME ASSESSMENT FOR DECISIONS PROGRAMME MONITORING AND EVALUATION 3

4 Programme Assessment Implementation Conceptualising And Designing Components/Stages of Programme Evaluation Needs Assessment What is the problem? Programme Theory Assessment How, in theory, does the programme fix the problem? Process Evaluation Does the programme work as planned? Impact Evaluation Were its goals achieved? The magnitude? Cost Effectiveness Given magnitude and cost, how does it compare to alternatives? 4

5 NEEDS ASSESSMENT Identifying the problem

6 Needs Assessment helps us understand the context and problem Does the problem we proposing to solve actually exist? What is the likely source of the problem? Of the solutions proposed and tried, why are they failing? What groups in the population should be targeted (Who is in most need?) What problems and opportunities does the target group face What challenges remain unaddressed Needs assessment basis of good programme design 6

7 Needs assessment can make us rethink on the design of our programme 1. No real problem 2. Problem addressed not a local priority 3. Cause of problem different from that assumed 4. Basic conditions needed to make programme effective not in place 7

8 Case study: Incentives for Immunisation Needs Assessment 8

9 Case study: Needs Assessment Every year, between 2 and 3 million people die from vaccine-preventable diseases Only 54% of 1-2 year olds in India receive the basic package of immunisations BCG, DPT, Polio, and Measles vaccines At least 5 visits In India, immunisations are offered for free but the immunisation rate remains low In rural Rajasthan, this rate falls to 22%..what is the likely source? 9

10 Underlying issues Supply side constraints: Average household is within 2 kilometers of the nearest clinic High absenteeism at government health facilities 45% of Auxiliary Nurse Midwives are absent on any given workday Demand side constraints: Cultural resistance, distrust in public health institutions People don t value immunisations: short-term cost for longterm (and invisible) benefits Limited income: parents can t afford to take a day off 10

11 Programme Theory Assessment Theory of Change: Blueprint for Change

12 Theory of Change - How will the programme, in theory, fix the problems identified? What are the prerequisites to meet the needs? How and why are those requirements currently lacking or failing? How does the programme intend to target or circumvent shortcomings? What services will be offered? How will the programme bring about change? 12

13 Case study: Incentives for Immunisation Programme Theory Assessment 13

14 Identifying potential solutions and designing programme Supply-side Solutions Strengthening of the existing government programme Stronger mechanisms to address staff absenteeism in clinics Tracking of unimmunised children and providing immunisation More regular camps? Demand-side Solutions Information campaigns Address lingering doubts regarding problems arising out of immunisations What about giving people incentives? Is this feasible? 14

15 How does the programme in theory fix the problem? GOAL Incentives for immunisation Immunisation Camps I N C R E A S E D I M M U N I S A T I O N Situation/Context Analysis: High health worker absenteeism, low value of immunisation, limited income and time 15

16 Programme s Theory of Change INPUT OUTPUT Resources for Immunisation Camps Camps are reliably open OUTCOME Parents bring children to the camps Incentives for Immunisation Incentives are delivered Parents bring children to the camps repeatedly GOAL Full Immunisation Situation/Context Analysis: High health worker absenteeism, low value of immunisation, limited income and time 16

17 PROCESS EVALUATION Making the programme work

18 Process Evaluation - Was the programme carried out as planned? Is it functioning well? Are basic tasks being completed? Is the intervention reaching the target population at the appropriate time? Is the intervention being completed well or efficiently and to the beneficiaries satisfaction? Are programme staff well informed and working hard? Is all the money accounted for? Process evaluation necessary for all programmes 18

19 Case study: Incentives for Immunisation Process Evaluation 19

20 Process Evaluation Establish camp Hiring nurses and administrators Installing temporary camp site Procuring vaccines and other medical supplies Organise incentive scheme Identify viable incentive Purchase kilos and dinner plate sets Demand-side Do parents visit the camps? Do they come back? 20

21 IMPACT EVALUATION Measuring how well it worked

22 Impact Evaluation Questions answered by impact evaluations Impact evaluations determines if a programme creates a change in an outcome(s) 1. What is the impact of a programme? i.e., specific intervention or a package of interventions 2. Which elements of the programme matter the most? 3. Which of the two alternative strategies should we pursue? 22

23 Which one of these would make a good question for an impact evaluation? A. What percentage of 3 year old children in Rajasthan were not fully immunised? B. What is the correlation between regular immunisation camps and immunisation rates? C. Does holding regular immunisation camps and providing incentives to parents improve immunisation rates of children? 23

24 Case study: Incentives for Immunisation Impact Evaluation 24

25 Randomisation design Not in evaluation (0) Camps (30) Total Population (700+ villages) Target Population (134) Evaluation Sample (134) Camps + Incentives (30) Random Assignment Control (74) 25

26 What was the impact? Immunisation rates 38% 17% 6% Control Camps Camps + Lentils 26

27 A good randomised evaluation should answer descriptive and process questions 27 Low immunisation rates Needs Assessment Intervention Intervention Inputs Black box? No increase in full immunisation Final outcome

28 A good randomised evaluation should answer descriptive & process questions Implementation failure Resources for Immunisation Camps Camps are reliably open Incentives for Immunisation Incentives are delivered Parents bring children to the camps Parents bring children to the camps repeatedly Full Immunisation Parents bring children to the camps Parents bring children to the camps repeatedly Full Immunisation Theory failure Resources for Immunisation Camps Camps are reliably open Incentives for Immunisation Incentives are delivered 28

29 COST-EFFECTIVENESS ANALYSIS Evidence-Based Policymaking

30 Cost effectiveness Costs per fully Immunised Child Rs Cost of Incentive Cost of Camp Rs Immunization Camps Rs 730 Camps + Incentives 30

31 Impact Evaluation questions What makes a good impact evaluation question? 31

32 Which one of these would make a good question for a needs assessment? A. What share of kids in Tanzania drop out of school before completing primary? B. Will providing kids with deworming pills or school uniforms do a better job of keeping kids in school? C. What share of students in government primary schools in Nepal received textbooks? 32

33 Which one of these would make a good question for an impact evaluation? A. Are agricultural extension agents giving farmers the same information they were trained on? B. What share of farmers in Kenya currently live on less than $2 a day? C. Which kind of fertilizer works best for a plot of maize? 33

34 Thank you Shubhra Mittal