PKH Evidence & Policy Implications: summary of Results from Impact Evaluation, Operations Analysis, and Spot Checks

Transcription

1 Investing in Indonesia s Institutions for Inclusive and Sustainable Development PKH Evidence & Policy Implications: summary of Results from Impact Evaluation, Operations Analysis, and Spot Checks Diskusi Pokja Kebijakan Monev TNP2K 29 March 2011

2 Today s topics Intro Methods Poverty Health Educ Labor Consumption Policy Summary of Results from PKH Impact Evaluation Methodology Poverty Health Education Child Labor Consumption Further evidence from Spot Checks and SMERU qualitative study Policy going forward: discussion Education Supply readiness Facilitators Affiliated providers 1 Kebijakan Monev 1

3 PKH was randomized to kecamatan, non-random to households. Intro Methods Poverty Health Educ Labor Consumption Policy Randomization is effective at ensuring that treatment decisions and response to treatment are orthogonal and uncorrelated. to which should be added including unobservable characteristics and unmeasured trends With equivalent distributions of characteristics we certain that treated and untreated groups are valid counterfactuals for each other: Equivalent distributions of characteristics are tested with baseline (pre-program) data on observable characteristics. 2 Kebijakan Monev 2

4 PKH was randomized to kecamatan, non-random to households. Intro Methods Poverty Health Educ Labor Consumption Policy The PKH kecamatan and household-selection processes: 3 Kebijakan Monev 3

5 PKH was randomized to kecamatan, but not 100% according to plan Intro Methods Poverty Health Educ Labor Consumption Policy Beware #1: the PKH kecamatan selection process was contaminated: 39 out of 180 (22%) surveyed lottery control kecamatan were found with PKH 2 out of 180 (1%) surveyed lottery treatment kecamatan were found to never have received PKH 4 Kebijakan Monev 4

6 PKH was non-random to households in an observable way Intro Methods Poverty Health Educ Labor Consumption Policy Beware #2: the household selection processes was purposive: 5 Kebijakan Monev 5

7 PKH was non-random to households in an observable way Intro Methods Poverty Health Educ Labor Consumption Policy Beware #2: the household selection processes was purposive: Result #1: PKH households worse off compared to even other eligible households Younger, larger, female-headed, in agriculture Less education, fewer assets, more likely to be BLT and Jamkesmas beneficiaries (but scholarships don t matter) Less connected to village authorities and less involved in governmental groups 6 Kebijakan Monev 6

8 Fixes for kecamatan contamination and non-random household selections Intro Methods Poverty Health Educ Labor Consumption Policy To continue using full matched-at-baseline sample while incorporating lottery/actual mismatches: Use lottery assignment as instrument for actual status Control (probabalistically) for those who received and missed incorrectly To compare treated households to like households in control kecamatan Use indices reflecting likelihood of receiving PKH; build indices from observed characteristics Control (probabalisitically) for those who would have received had PKH been in their kec To provide an alternative specification (robustness) Difference-in-differences method with two alternatives (xsection or matched panel) Provide estimates of behaviors over randomly-receiving or randomly not-receiving hh only What difference in differences (DID) does: subtract from pre- to post-pkh changes the same over-time change in non-pkh areas. 7 Kebijakan Monev 7

9 PKH generated significant reductions in poverty, poverty gap measures Intro Methods Poverty Health Educ Labor Consumption Policy 1 Implied reductions in poverty measures Average baseline (2007) expenditure (household average, per capita per month): IDR 199,000 average for all eligible households; eventual PKH households slightly poorer at baseline Average PKH household increase in expenditure (2007 to 2009): IDR 19,000 (9.5% of baseline) this is over and above what similar non-pkh families in control kecamatan experienced. PKH was enough to move ~700,000 eligible beneficiaries out of poverty (on average) March 2009 poverty line: IDR 200,262 PKH share in overall measured poverty reduction could have been 82% (on average) : : headcount poverty rate 16.6% 14.1% implies 858,000 fewer poor hh Average baseline expenditure (2007) for eventual PKH households : IDR 184,000 Average PKH impact on these households (IDR 19,000) = 10% of baseline PKH moved ~700,000 actual beneficiaries to the March 2009 poverty line implicit reduction in poverty gap/severity: 1.9/0.74 index points PKH share in decrease in poverty gap/severity could have been 73%/65% (on average): : actual poverty gap and actual severity Note: all results approximately the same under the assumption that the full value of an average 2008 PKH transfer (IDR 1.2 million) is used (by a PKH household with an average of 5.2 members) on expenditure items used to calculate total household expenditure. Such an assumption implies an average increase in expenditure of IDR 19,231 per-capita per month. Kebijakan Monev 8

10 Many Healthy Behaviours saw significant and positive increases Intro Methods Poverty Health Educ Labor Consumption Policy 2a Significant Health Behaviour Changes 9 Kebijakan Monev 9

11 Many Healthy Behaviours saw significant and positive increases Intro Methods Poverty Health Educ Labor Consumption Policy 2a Significant Health Behaviour Changes 10 Kebijakan Monev 10

12 Many Healthy Behaviours saw significant and positive increases Intro Methods Poverty Health Educ Labor Consumption Policy 2a Significant Health Behaviour Changes 11 Kebijakan Monev 11

13 Many Healthy Behaviours saw significant and positive increases Intro Methods Poverty Health Educ Labor Consumption Policy 2a Significant Health Behaviour Changes Spillovers to non-pkh hhs 12 Kebijakan Monev 12

14 Many Healthy Behaviours saw significant and positive increases Intro Methods Poverty Health Educ Labor Consumption Policy 2a Significant Health Behaviour Changes 13 Kebijakan Monev 13

15 Healthy Behaviours benefitted from significant promotion Intro Methods Poverty Health Educ Labor Consumption Policy 2a Significant Health Behaviour Changes further evidence SMERU Qual Study - Java Barat and NTT; 24 desa/kelurahan); Java Barat accessible; NTT remote For maternal & child care (MCH), PKH led to Posyandu attendance (primarily NTT) attendance at Posyandu alternatives (primarily urban) awareness of and activity on MCH issues both among beneficiaries and village officials, midwives, posyandu cadres, community leaders, religious leaders, NGOs Facilitators crucial: positive encouragement and negative threats PKH did not coincide with or lead to access to MCH for remote villagers Factors mentioned: no incentives and/or logistical support for posyandu cadres; absent midwives; poor roads and rainy-season shutdown; lack of public transport community involvement in MCH budget or provision decisions 14 Kebijakan Monev 14

16 Not all Healthy Behaviours saw significant increases for PKH hhs Intro Methods Poverty Health Educ Labor Consumption Policy 2b Not Significant Health Behaviour Changes 15 Kebijakan Monev 15

17 Provider quality and remoteness contributed to remaining gaps in health Intro Methods Poverty Health Educ Labor Consumption Policy 2b Not Significant Health Behaviour Changes further evidence CHRUI Spot Checks - Java (East, West, Jakarta), South Kalimantan, NTB; 36 kecamatan; 1800 mothers, 108 facilitators, 108 schools, 36 Puskemas, PKH did not coincide with or lead to access to puskemas services for remote villagers Factors mentioned: distance, bad transport, no transport available access to comprehensive service at posyandu (materials and tools; information and copies of records; complete schedule of treatments/interventions) fees at puskesmas grievance redress when services rejected or delayed by health service providers. full availability of supplies at health service providers Human capital: dentists, midwives, public health officer, paramedics, nutrionist, pharmacist not always available at puskemsas Vaccines& Vaccine storage: occasionally missing at puskemsas 16 Kebijakan Monev 16

18 Provider quality and remoteness contributed to remaining gaps in health Intro Methods Poverty Health Educ Labor Consumption Policy 2b Not Significant Health Behaviour Changes evidence in urban or densely populated Java 17 Kebijakan Monev 17

19 Medium- to long-term health outcomes are mostly showing no changes yet Intro Methods Poverty Health Educ Labor Consumption Policy 2c Mostly Not Significant Health Outcomes 18 Kebijakan Monev 18

20 But health-seeking behavior has already changed Intro Methods Poverty Health Educ Labor Consumption Policy 2c Mostly Not Significant Health Outcomes evidence from urban, dense, and agricultural areas 19 Kebijakan Monev 19

21 High baseline levels in Education make further increases difficult Intro Methods Poverty Health Educ Labor Consumption Policy 3 Mostly Not Significant Education Outcomes 20 Kebijakan Monev 20

22 Low levels of Wage work make further reductions difficult Intro Methods Poverty Health Educ Labor Consumption Policy 4 Mostly Not Significant Child Labor Outcomes 21 Kebijakan Monev 21

23 Education supply remained inadequate; advocacy was insufficient Intro Methods Poverty Health Educ Labor Consumption Policy 3 Insignificant Education Changes further evidence SMERU Qualitative In education, PKH led to Awareness among beneficiaries and village officials, midwives, posyandu cadres, community leaders, religious leaders, NGOs Awareness of links from education to employers requirements and better future For primary education Enrollment did not increase because of previously high levels of attendance Facilitators crucial for in Hours/Attendance: positive encouragement and negative threats note: SMERU finds more effective facilitators in NTT b/c of reduced range, not reduced caseload For secondary education Enrollment increased only in NTT (again, facilitators crucial) Facilitators crucial for in Hours/Attendance PKH did not coincide with or lead to access to 2 nd ary schools Factors mentioned: remoteness/distance to schools; lack of public trans; overall cost; early marraige/pregnancy; job demands (harvest) supply of quality at 2 nd ary schools Factors mentioned: overcapacity (from BOS); decline in parents participation (again from BOS); low levels of teacher education; lack of facilities (libraries, labs, teaching aids, electricity, clean water) 22 Kebijakan Monev 22

24 Education supply remained inadequate; advocacy was insufficient Intro Methods Poverty Health Educ Labor Consumption Policy 3 Insignificant Education Changes further evidence CHRUI Spot Checks PKH did not coincide with or lead to participation or attendance Factors mentioned: attendance rates already approx. 100%.; no outreach to those who leave school early; PKH did not cover opportunity costs of secondary school access to 2 nd ary schools Factors mentioned: remoteness/distance to schools (2km/22min on average, 20km/120min maximum to reach with foot or pedal power as the most common mode of transport); no tuition waivers; supply of quality at 2 nd ary schools Factors mentioned: low levels of teacher education; overcrowding; lack of facilities (libraries, labs, teaching aids, electricity, clean water) 23 Kebijakan Monev 23

25 Region & Sector trends are encouraging; household trends less so Intro Methods Poverty Health Educ Labor Consumption Policy 3,4 mostly not significant Education & Child Labor further evidence 24 Kebijakan Monev 24

26 Consumption, especially of health goods and services, increases Intro Methods Poverty Health Educ Labor Consumption Policy 5 Significant Consumption Increases 25 Kebijakan Monev 25

27 Consumption, especially of health goods and services, increases Intro Methods Poverty Health Educ Labor Consumption Policy 5 Consumption demonstrates significant increases in health seeking behaviour note: does absence of change in education spending indicate anything re: hh constraints to education? 26 Kebijakan Monev 26

28 PKH benefits from attention; remaining reforms become clear Intro Methods Poverty Health Educ Labor Consumption Policy Education are PKH benefits enough to expect improvements given costs? SMP/SMA costs per year for poor hh: ~IDR 2.5 million or ~30% of expenditure; PKH households generally did not receive scholarships PKH family size: 5.2 members, 3-4 children needing care 27 Kebijakan Monev 27

29 PKH benefits from attention; remaining reforms become clear Intro Methods Poverty Health Educ Labor Consumption Policy Education or is it payment timing and program design? Education Attained by Indonesian Age 26-28, Year Graduated Graduated 6 SD/MI 7 8 9SMP/MTs 10 Years of Education quintile 1 quintile 2 quintile 3 quintile 4 quintile 5 In the beginning, the 4-per year payment design was changed to 3-per year. As a consequence, there was no PKH payment synchronized with due dates for school fees. These modifications were partially due to the inability of UP-PKH and PT Pos to manage keep the validation and verification process timely (throughput of forms and data) 28 Kebijakan Monev 28

30 PKH benefits from attention; remaining reforms become clear Intro Methods Poverty Health Educ Labor Consumption Policy Education or is it in implementation? Socialization to affiliated providers was weak Providers may not have understood that conditionalities (and compliance monitoring) wasn t important and the PKH was not just business as usual for them See process engineering assessment and CHRUI for factors behind weak socialization activities Facilitators were not as effective at encouraging education Facilitators and the community generally did not make effort to re-insert drop-outs into the system; facilitators were sometimes too diffuse to be able to visit remote households as soon as drop-outs were noted See CHRUI for factors behind inability of facilitators to improve education behaviors where they were too disperse Affiliated agencies did not isolate PKH households for monitoring or assistance PKH households generally did not receive scholarships or subsidized transportation In reality, all of these issues are at work and all need serious thought Weakness in one area (form processing) has knock-on effects in other areas (altered payment schedule) which can affect outcomes (inability to recapture drop-outs or improve transition rates) 29 Kebijakan Monev 29

31 Relative to other Indonesia SSN, PKH expenditures appear appropriate Intro Methods Poverty Health Educ Labor Consumption Policy 1 PKH expenditures show appropriate shares for benefits relative to support operations WB Social Assistance Public Expenditure Review 30 Kebijakan Monev 30

32 Relative to other Indonesia SSN, PKH expenditures appear appropriate Intro Methods Poverty Health Educ Labor Consumption Policy 1 PKH expenditures show appropriate shares for benefits relative to support operations WB Social Assistance Public Expenditure Review PKH s operation costs have come down, similar to international examples Other facilitated programs spending similar amounts on operations Other programs with similar coverage or similar benefits (Jamkesmas, Scholarships) spend very little on non-benefit operations These programs are not facilitated These programs do not reach only poor households These programs have larger impacts for non-poor households. These programs rely on local-level administration (at the facility) to run the program they are essentially transfers from central government agencies to service providers. 31 Kebijakan Monev 31

33 Qualitative Studies raised several additional operational issues Intro Methods Poverty Health Educ Labor Consumption Policy 2 Validation and updates to household rosters CHRUI Spot Checks 18% households could not be re-verified; 69% had Jamkesmas; only 18% had Scholarships; mismatches between UP-PKH pusat databases and actual recipients; 84% households reported demographic changes; updates were not monthly everywhere Factors mentioned: lack of personnel; lack of forms; power outages; nonfunctioning MIS (including automatic checks by province or central level); inaccurate BPS data Solutions uji coba (pilot phase): Targeting improvements and recertification strategies; MIS fully functional (and staffed?) everywhere; Form printing and delivery supply chain improvements Building in data sweeps for mismatches to occur some weeks before payment process begins 32 Kebijakan Monev 32

34 Qualitative Studies raised several additional operational issues Intro Methods Poverty Health Educ Labor Consumption Policy 3 Compliance verification and conditioning CHRUI Spot Checks 71%, 68%, 29%, 24%, 19% with BCG, measles, polio, DPT, and hepatitis vaccines (respectively); 2.2% not going to school; and no recorded changes in benefits. Factors mentioned: mismatch between names of mothers/children reported at local level and in UP-PKH pusat database; forms content, printing, and delivery problematic; lack of incentives (mainstreaming); lack of socialization/enforcement on conditionalities (including to affiliated operators like PT Pos) Solutions sudah: MIS fully functional nearly everywhere; CVS system operational for 85% of hh (@ Jan 2010, increasing lately) uji coba (pilot phase): Form printing and delivery supply chain improvements (joint with PT Pos and facilitators) Resocialization and re-examining socialization schedule; audit of socialization process. sharing lessons with affiliated providers (MoNE, MoRA, MoH); incentive schemes or enforceable MOUs with affiliated providers? 33 Kebijakan Monev 33

35 Qualitative Studies raised several additional operational issues Intro Methods Poverty Health Educ Labor Consumption Policy 4 Payment timeliness, scheduling, and payment processing schedules CHRUI Spot Checks 75% report correct payment amounts and 95% received exactly the receipt amount; some districts scheduled 2 or 3 tranches per year instead of 4. Factors mentioned: CVS and data updates not operating; data mismatch. Solutions sudah : 4 per year schedule enforced; timing tranches to school fees schedule; MIS fully functional nearly everywhere uji coba (pilot phase): Form printing and delivery supply chain improvements; CVS system operational for 85% of hh (@ Jan 2010) 34 Kebijakan Monev 34

36 Process Engineering examines in detail the PKH machine Intro Methods Poverty Health Educ Labor Consumption Policy 5 Socialization & Complaints and Grievances systems Ayala Consulting (& confirmed in CHRUI Spot checks) Socialization delegated to KemenKomInfo not adequately carried out (to beneficiaries and affiliated providers) either in content, frequency, or intensity; no printed materials; officers not able to adequately answer questions; lack of monitoring of socialization process resulted in lingering misunderstanding; PKH beneficiaries not thought elibible for Raskin or Jamkesmas; most info via word of mouth or media reports of malfeasance Majority of beneficiaries did not know how to file complaint and were unaware of the CGS; written complaints too costly (fear of losing benefits or faux pas); lack of forms; no response on the small amount of complaints received; MIS system does not automatically respond to incoming grievances or complaints Potential Solutions uji coba (pilot phase): recertification and waiting lists in the targeting exercise 35 Kebijakan Monev 35

37 Process Engineering examines in detail the PKH machine Intro Methods Poverty Health Educ Labor Consumption Policy 5 Socialization & Complaints and Grievances systems Potential Solutions (continued) DepSos, Bappenas, or TNP2K recapture socialization mandate in order to deliver better socialization, policy, materials & effort Case Management system: facilitators as one-stop shop for all social assistance initiatives for eligible households (April re-visit by int l consultant) Affiliated agency MOUs: positive and negative incentives to generate interest and effectiveness in affiliated service providers (PT Pos, KemenKomInfo, MoNE, MoRA, MoH) (April re-visit by int l consultant) Review of current CGS procedures in manual and investigation of nondeployment (April re-visit by int l consultant) 36 Kebijakan Monev 36

38 Process Engineering examines in detail the PKH machine Intro Methods Poverty Health Educ Labor Consumption Policy 6 UP-PKH organization and cross-agency organization is ineffective Ayala Consulting & int l experts (& confirmed in CHRUI Spot checks) Affiliated agencies and local gov ts agencies consider PKH a burden and do not instruct regional offices to give continuous support; coordination meetings not attended; DepSos lacks authority and/or appropriate agreements to monitor progress (and remedy lack of progress) in affiliated agencies (in 2010, no PKHrelated activities were undertaken by KemenKomInfo until earliest August); MOUs inappropriate and unenforceable regional coordinators stuck in Jakarta; DepSos lacks capacity/authority to implement a full-scale continuous reform cycle (improvements at the margin); no performance evaluations carried out by UP-PKH; no TA or monitoring available at/from UP-PKH propinsi level (this is inadequate for a country as large and diverse as Indonesia); unclear responsibilities for regional and provincial coordinators; PKH has no specialized personnel for policy/planning, training, interagency coordination, M&E and continuous reform. Potential Solutions sudah : Korwil moved back to province 37 Kebijakan Monev 37

39 Process Engineering examines in detail the PKH machine Intro Methods Poverty Health Educ Labor Consumption Policy 6 UP-PKH organization and cross-agency organization is ineffective Potential Solutions (continued) Affiliated agency MOUs: positive and negative incentives to generate interest and effectiveness in affiliated service providers (PT Pos, KemenKomInfo, MoNE, MoRA, MoH) (April re-visit by int l consultant) Review of current UP-PKH organization and feasible blueprint improvements (April re-visit by int l consultant) TNP2K to raise PKH s profile; point out interest other agencies may have; point out the wealth of data indicating weaknesses in cross-agency collaboration 38 Kebijakan Monev 38

40 Process Engineering examines in detail the PKH machine Intro Methods Poverty Health Educ Labor Consumption Policy Issue #6: Organization within UP-PKH Placeholder for Ayala s current 39 Kebijakan Monev 39

41 Investing in Indonesia s Institutions for Inclusive and Sustainable Development PKH Evidence & Policy Implications: summary of Results from Impact Evaluation, Operations Analysis, and Spot Checks Diskusi Pokja Kebijakan Monev TNP2K 29 March 2011