The World Bank IMPLEMENTATION COMPLETION AND RESULTS REPORT (TF014687) A GRANT IN THE AMOUNT OF USD 1.92 MILLION

Transcription

1 Public Disclosure Authorized The World Bank Report No: ICR Public Disclosure Authorized IMPLEMENTATION COMPLETION AND RESULTS REPORT (TF014687) ON A GRANT IN THE AMOUNT OF USD 1.92 MILLION Public Disclosure Authorized TO YAYASAN KELOLA FOR THE PNPM CREATIVE COMMUNITIES II PROJECT September 21, 2015 Public Disclosure Authorized Indonesia Social Development Unit Urban, Rural & Social Development East Asia and Pacific Region

2 CURRENCY EQUIVALENTS (Exchange Rate Effective March 31, 2015) Currency Unit = Rupiah USD 1.00 = IDR 13,075 FISCAL YEAR January 1 December 31 ABBREVIATIONS AND ACRONYMS APBDes BAPPENAS BETF CBO CCG CCI CCII CDD CEP CPS CSO DFAT DPR DO EIRR FM Generasi GoI GRM IBRD ICM ICR ID IDR IFR IRI ISR IGSES JMC JSDF Kelola Anggaran Pembangunan dan Belanja Desa (Village Budget) Badan Perencanaan Pembangunan Nasional (National Development Planning Board) Bank-Executed Trust Fund Community Based Organization Creative Community Groups Creative Communities I Creative Communities II Community-Driven Development Creative Empowerment Process Country Partnership Strategy Civil Society Organization Department of Foreign Affairs and Trade (Government of Australia) Dewan Perwakilan Rakyat (People s Representative Assembly) Development Objective Economic Internal Rate of Return Financial Management PNPM Generasi Sehat dan Cerdas (PNPM Healthy and Smart Generation) Government of Indonesia Grant Reporting and Monitoring Report International Bank for Reconstruction and Development Implementation Completion Memorandum Implementation Completion and Results Identification Indonesian Rupiah Interim Financial Report Intermediate Results Indicator Implementation Status and Results Implementation Guidelines for Social and Environmental Safeguards Joint Management Committee Japan Social Development Fund Yayasan Kelola (Kelola Foundation) i

3 KDP KMN KPI KPK KPMD M&E MC MIS MoHA MoV NGO NMC O&M PAD PCR PEKKA PDO PMD PNPM PP PPP PSF PTO QEA QSA RETF RMC SOP TfD TPK TTL VL WB Kecamatan Development Project Konsultan Manajemen Nasional (National Management Consultant) Key Performance Indicator Kader Pemberdayaan Kreatif (CCII Creative Empowerment Cadre) Kader Pemberdayaan Masyarakat Desa (PNPM Village Empowerment Cadre) Monitoring and Evaluation Managing Contractor Management Information System Ministry of Home Affairs Ministry of Villages, Disadvantaged Areas and Transmigration Non-Government Organization National Management Consultant Operations and Maintenance Project Appraisal Document Project Completion Report Pemberdayaan Perempuan Kepala Keluarga (Female-Headed Family Empowerment Program) Project Development Objective Pemberdayaan Masyarakat Desa (Village Community Empowerment), Ministry of Home Affairs Program Nasional Pemberdayaan Masyarakat (National Community Empowerment Program) Project Paper Purchasing Power Parity PNPM Support Facility Petunjuk Teknis Operasional (Technical Operational Guidelines) Quality at Entry Assessment Quality of Supervision Assessment Recipient-Executed Trust Fund Regional Management Consultant Standard Operating Procedure Theater for Development Tim Pengelola Kegiatan (Implementating Team of PNPM) Task Team Leader Village Law World Bank Regional Vice President: Country Director: Senior Global Practice Director Acting Program Manager: Project Team Leader: ICR Lead Author: Axel van Trotsenburg Rodrigo A. Chaves Ede Jorge Ijjasz-Vasquez Kevin Tomlinson Natasha Hayward Jacqueline Pomeroy ii

4

5 INDONESIA SMALL GRANT TO YAYASAN KELOLA FOR THE PNPM CREATIVE COMMUNITIES II PROJECT CONTENTS Data Sheet A. Basic Information... iiv B. Key Dates... iiv C. Ratings Summary... iiv D. Sector and Theme Codes... v E. Bank Staff... v F. Results Framework Analysis... v G. Ratings of Project Performance in ISRs... iix H. Restructuring (if any)... iix I. Disbursement Profile... x 1. Project Context, Development Objectives and Design Key Factors Affecting Implementation and Outcomes Assessment of Outcomes Assessment of Risk to Development Outcome Assessment of Bank and Borrower Performance Lessons Learned Comments on Issues Raised by Borrower/Implementing Agencies/Donors Annex 1. Project Costs and Financing Annex 2. Outputs by Component Annex 3. Economic and Technical Analysis Annex 4. Loan Preparation and Implementation Support/Supervision Processes Annex 5. Beneficiary Perceptions Survey Annex 6. Stakeholder Workshop Report and Results Annex 7. Borrower's Completion Report Annex 8. List of Supporting Documents MAP IBRD33420 iii

6

7 A. Basic Information Country: Indonesia Project Name: PNPM Creative Communities II Project ID: P L/C/TF Number(s): TF ICR Date: 09/23/2015 ICR Type: Core ICR Lending Instrument: TAL Grantee: Original Total Commitment: USD 0.61M Disbursed Amount: USD 1.69M Revised Amount: USD 1.92M Environmental Category: C Implementing Agencies: Kelola Foundation Cofinanciers and Other External Partners: Directorate General of Village Community Empowerment, Ministry of Home Affairs (MOHA) B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 01/18/2013 Effectiveness: Appraisal: 02/28/2013 Restructuring(s): Approval: 04/24/2013 Mid-term Review: 11/30/ /11/2014 Closing: 12/31/ /31/2015 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Risk to Development Outcome: Bank Performance: Grantee Performance: Moderately Satisfactory Moderate Moderately Satisfactory Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Satisfactory Quality of Supervision: Moderately Satisfactory Implementing Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Performance: Performance: iv Satisfactory Satisfactory

8 C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Performance (if any) Potential Problem Project at any time (Yes/No): Problem Project at any time (Yes/No): DO rating before Closing/Inactive status: No No Satisfactory Quality at Entry (QEA): Quality of Supervision (QSA): None None Rating D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Vocational training Theme Code (as % of total Bank financing) Participation and civic engagement Social Inclusion E. Bank Staff Positions At ICR At Approval Vice President: Axel van Trotsenburg Axel van Trotsenburg Country Director: Rodrigo A. Chaves Stefan G. Koeberle Practice Manager/Manager: Kevin A Tomlinson Jan Weetjens Project Team Leader: Natasha Hayward Susanne Holste ICR Team Leader: ICR Primary Author: Natasha Hayward Jacqueline L. Pomeroy F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) Improve the ability of villagers, especially poor and marginalized people, to participate actively in PNPM processes using cultural expressions in project locations. Revised Project Development Objectives (as approved by original approving authority) The Project Development Objectives did not change over the course of the project. v

9 (a) PDO Indicator(s) Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Indicator 1: Increase in public participation in village planning processes in project locations Value % quantitative or 50% 50% (Phase 1) qualitative) Date achieved 12/31/ /31/ /31/2015 Phase 1 villages (39% of expenditures) experienced increases of %. Due Comments (incl. % achievement) to early closure of PNPM activities by GoI, there were no planning processes in Phase 1 Extension villages (61% of expenditures). 70% of beneficiaries in Phase 1 Extension said they were more motivated to participate in formal village meetings after the project. Indicator 2: Increased participation of women in village planning processes Value quantitative or qualitative) 50% 50% 93% (Phase 1) Date achieved 12/31/ /31/ /31/2015 Women s participation increased by 93% in Phase 1 locations. Phase 1 Comments Extension had no planning processes due to early closure of PNPM activities by (incl. % GoI, but 52% of beneficiaries were women and 70% of beneficiaries said they achievement) were more motivated to participate in formal village meetings after the project. Indicator 3: Increased participation of marginalized people in village planning processes Value quantitative or 50% N.A. qualitative) Date achieved 03/31/ /31/2015 Comments (incl. % achievement) Indicator 4: Value quantitative or Qualitative) Phase 1 evaluation data did not identify marginalized people. Phase 1 Extension had no PNPM planning processes due to early closure of PNPM activities by GoI, but 72% of beneficiaries were marginalized and 70% of beneficiaries said they were more motivated to participate in formal village meetings after the project. Increased participation of poor community members in village planning processes 50% 50% 268% (Phase 1) Date achieved 12/31/ /31/ /31/2015 Participation of the poor increased by 268% in Phase 1 locations. Phase 1 Comments Extension had no planning processes due to early closure of PNPM activities by (incl. % GoI, but 45% of beneficiaries were poor and 70% of beneficiaries said they were achievement) motivated to participate in formal village meetings. Beneficiaries have increased understanding of PNPM and village planning Indicator 5: process Value 0 70% 70% 53% vi

10 Indicator Baseline Value Original Target Values (from approval documents) vii Formally Revised Target Values Actual Value Achieved at Completion or Target Years quantitative or qualitative) Date achieved 05/30/ /31/ /31/ /31/2015 Comments (incl. % achievement) Indicator 6: Value quantitative or Qualitative) 53% of Phase 1 Extension beneficiaries reported increased understanding of PNPM and village planning processes, with another 28% stating I don t know possibly reflecting confusion over the ending of PNPM and uncertainty over processes under the Village Law. Women beneficiaries with increased PNPM understanding 0 50% 50% 55% Date achieved 05/30/ /31/ /31/ /31/ % of Phase 1 Extension women beneficiaries reported increased Comments understanding of PNPM and village planning processes. Overall, an average (incl. % 28% stated I don t know possibly reflecting confusion over the ending of achievement) PNPM and uncertainty over processes under the Village Law. Indicator 7: Youth beneficiaries with increased PNPM understanding Value quantitative or qualitative) 0 50% 42% Date achieved 05/30/ /31/ /31/2015 Comments (incl. % achievement) Indicator 8: Value quantitative or Qualitative) 42% of Phase 1 Extension youth beneficiaries reported increased understanding of PNPM and village planning processes. Overall, an average 28% stated I don t know possibly reflecting confusion over the ending of PNPM and uncertainty over processes under the Village Law. Marginalized beneficiaries with increased PNPM understanding 0 50% 50% 40% Date achieved 05/30/ /31/ /31/ /31/ % of Phase 1 Extension marginalized beneficiaries reported increased Comments understanding of PNPM and village planning processes. Overall, an average (incl. % 28% stated I don t know possibly reflecting confusion over the ending of achievement) PNPM and uncertainty over processes under the Village Law. Indicator 9: Poor beneficiaries with increased PNPM understanding Value quantitative or qualitative) 0 50% 50% 45% Date achieved 05/30/ /31/ /31/ /31/2015 Comments (incl. % achievement) 45% of Phase 1 Extension poor beneficiaries reported increased understanding of PNPM and village planning processes. Overall, an average 28% stated I don t know possibly reflecting confusion over the ending of PNPM and uncertainty over processes under the Village Law. Indicator 10: Number of people trained

11 Indicator Baseline Value Original Target Values (from approval documents) viii Formally Revised Target Values Actual Value Achieved at Completion or Target Years Value quantitative or qualitative) Date achieved 05/30/ /31/ /31/2015 Comments (incl. % achievement) 22 PNPM facilitators, 31 CCII facilitators, 24 CCII village Creative Empowerment Cadres, and 1,300 village cultural workers. Indicator 11: Training module and tools for PNPM and Village Law implementation Value quantitative or qualitative) 0 Modules and tools developed 4 Date achieved 05/30/ /31/ /31/2015 Comments (incl. % achievement) Two manuals for the Creative Empowerment Process (one on theater for development and another on participatory video), one manual for effective facilitation using creative approaches, and one advanced module on sustaining growth and continuation of creative community groups. Indicator 12: Number of cultural workers Value quantitative or qualitative) Date achieved 05/30/ /31/ /31/2015 Comments (incl. % achievement) 1302 local cultural workers implemented cultural activities linked to village planning. The project attracted and mobilized far more cultural workers than expected at the village level to partake in its activities. (b) Intermediate Outcome Indicator(s) Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Indicator 1 : Project development completed and design modifications tested Value (quantitative Yes Yes or qualitative) Date achieved 12/31/ /31/2015 Comments Project manual developed and full project design finalized after initial testing (incl. % during Phase 1. achievement) Indicator 2 : Trainers are trained for implementing CC II Value (quantitative or qualitative) Date achieved 12/31/ /31/2015

12 Indicator Comments (incl. % achievement) Indicator 3 : Value (quantitative or qualitative) Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Phase 1: 72 (including 22 PNPM facilitators) Phase 1 Extension: 48 CC II facilitators based in Bali, East Nusa Tenggara and West Java. Management team and operational structure established for full implementation Yes Date achieved 12/31/ /31/2015 Comments (incl. % achievement) Achieved prior to commencement of Phase 1 Extension. Yes G. Ratings of Project Performance in ISRs No. Date ISR Archived DO IP Actual Disbursements (USD millions) 1 10/13/2013 Satisfactory Satisfactory /25/2014 Satisfactory Satisfactory /03/2014 Satisfactory Satisfactory /31/2015 Moderately Unsatisfactory Satisfactory 1.69 H. Restructuring (if any) Restructuring Date(s) Board Approved PDO Change ISR Ratings at Restructuring DO IP Amount Disbursed at Restructuring in USD millions Reason for Restructuring & Key Changes Made 11/29/ Extension of closing date 06/13/2014 S S 0.61 Extension of closing date 08/22/2014 S S 0.61 Extension of closing date and additional financing ix

13 I. Disbursement Profile x

14 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal Indonesia had seen a return to strong economic growth over the prior fifteen years, with poverty falling from more than 23 percent in 1999 to 12.5 percent in However, in addition to the 32.5 million households that still lived below the national poverty line (PPP$1.19 per day), roughly half the population was clustered just above the poverty line these people suffered from a low standard of living and remained highly vulnerable even to small shocks to income and consumption. Of the total population living in poverty, 65 percent were in rural areas where approximately half of the population still resides, where income and employment opportunities and delivery of basic services remain seriously constrained and additionally are characterized by regional disparities. Non-income poverty as reflected in health and education access and outcomes was then, and continues to be, a serious problem. Many of the country s institutions and local governments were not effective in delivering services in an accountable manner. These shortcomings often prevented the poorest from benefiting more fully from growth and poverty reduction efforts. To address issues of poverty, in 2007 the Government of Indonesia (GoI) launched the National Program for Community Empowerment (PNPM), building on its Kecamatan Development Program (KDP), a community-driven development (CDD) initiative in selected sub-districts or kecamatan. The umbrella PNPM covered KDP and other linked CDD activities and scaled up coverage substantially to cover all rural villages and urban wards in the country. The GoI s longer term PNPM Roadmap envisioned institutionalizing community participation, including of the poor and marginalized, into the local planning and service delivery processes. Ultimately, this was successfully enshrined in the Village Law (Law No. 6 of 2014), which promises significantly increased discretionary grant resources to village governments from 2015/16, and provides for citizen engagement in decision making and oversight processes although this was not in place at the time of project preparation for Creative Communities II. PNPM s quick national scale-up from 2007 strained the facilitator recruitment process, their performance management, oversight and training, and resulted in some challenges to the effectiveness of that facilitation, given the massive scale at which PNPM was implemented. Combined with some reported community fatigue from the repeated planning cycles, the possibility that enthusiasm for public participation could wane over time was recognized, including risks that community voice especially that of the poor and marginalized could diminish. The precursor to CCII, the Creative Communities Fund (CCI) was created in November 2007 in an effort to improve PNPM outcomes, by providing small grants to villagers in three provinces to conduct cultural activities that enhanced citizen participation and the community empowerment process. The current project, CCII, built on that experience by awarding a competitive grant to a civil society organization to develop and test the delivery of targeted cultural activities to diverse communities in an effort to strengthen the voice and participation of the poor and marginalized in village planning. CCII was proposed as a grant financing operation linked to PNPM Rural IV (IBRD Loan Number 80790). 1

15 Rationale for Bank Involvement. The GoI and the World Bank had a longstanding working and lending relationship through KDP 1 and subsequently PNPM, and had cooperated successfully on a number of smaller, more innovative local development programs. Moreover, the PNPM Support Facility (PSF) platform a multi-donor financed initiative, led by the Bank was designed to finance additional research and pilot activities around PNPM. The GoI recognized the important roles CSOs play in community empowerment, and shared the Bank s vision that PNPM was a broad-based program that benefited from multi-stakeholder participation and support, including working together with CSOs to improve PNPM s outcomes. PSF provided the platform to: (a) access targeted trust funds; and (b) assist the GoI in applying these resources indirectly to communities through supervised grants to a civil society organization (CSO). Consistency with Country Partnership Strategy. The community-driven development approach of PNPM was a key component of the Bank s Country Partnership Strategy (CPS) for , which was linked to the GoI s Master Plan for Acceleration and Expansion of Indonesia s Economic Development CCII was in line with the CPS s pro-poor agenda and through PNPM s CDD process aimed to: a) increase the ability of the poor to voice their needs and benefit through PNPM; and, b) enhance broad-based participation crucial to achieving PNPM s poverty reduction and community empowerment outcome. 1.2 Original Project Development Objective (PDO) and Key Performance Indicators The PDO of the Creative Communities II (CCII) project (World Bank, May 2013) is to improve the ability of villagers, especially poor and marginalized people, to participate actively in PNPM processes using cultural expressions in project locations. Outcome indicators The CCII project consisted of two phases: Phase 1: Operational Setup, including piloting activities in three sub-districts; and Phase 2: Implementation in 50 sub-districts 2. The Phase I KPIs for the outcome indicators during Phase I were identified in the PAD with targets to be quantified during Phase 1: 1. Increase in general public participation in PNPM Rural and Generasi 3 processes in project locations. a. Number of participants in consultative and decision making activities in PNPM Rural and Generasi processes. b. Percentage of female participants in consultative and decision making activities in PNPM Rural and Generasi processes. 2. Increased ability of poor and marginalized people to participate in PNPM Rural and Generasi processes in project locations. a. Percentage of participants in CCII activities who are poor and marginalized people. 1 The first CDD loan for KDP started in The planned Phase 2 was scaled down to 24 sub-districts in the Phase 1 Extension following project restructuring. 3 PNPM Generasi Sehat dan Cerdas (PNPM Healthy and Smart Generation) is a conditional cash transfer pilot, under the wider PNPM umbrella, now being scaled-up by the GoI focusing on improving health and education outcomes. 2

16 b. Percentage of female poor and marginalized participants in CCII activities. 3. Increased ability among PNPM facilitators and cultural groups in using cultural mediums to strengthen participation in PNPM Rural and Generasi processes in project locations. a. PNPM facilitators use cultural mediums to reach PNPM Rural and Generasi beneficiaries (percentage). b. CSOs that partner with PNPM Rural and Generasi in using cultural activities to increase participation in PNPM Rural and Generasi (number, types). 1.3 Revised PDO and Key Indicators The PDO remained unchanged throughout project implementation. The project was formally restructured three times with the first two restructurings changing only the closing date. The third and final restructuring (see section 1.7 for more detail) (World Bank, August 22, 2014) revised the KPIs by adding (i) the percentage of poor and (ii) the percentage of marginalized participants to indicators (1) and (2), as well as number of people trained to indicator (2). Indicator number three was modified as follows: 3. PNPM facilitators and cultural workers have increased capacity to use cultural mediums to improve participation in village planning a. Training module and tools developed for PNPM and Village Law implementation. b. Number of cultural workers who implement cultural activities to facilitate village planning. 1.4 Main Beneficiaries The main beneficiaries of the project consisted of the poor and marginalized inhabitants of the target sub-districts located in six provinces. They would benefit from participation in the cultural groups, participation in the consultations the groups conducted, and ultimately from community infrastructure and services resulting from PNPM and village planning processes that met their expressed needs. The secondary beneficiaries were the district, sub-district and village governments and facilitators in which the 24 target sub-districts were located, who benefited from improved capacity to broaden participation and strengthen the voices of the poor and marginalized in the community planning process, which ultimately would improve their ability to provide needed services. 1.5 Original Components Component I: Community-based Cultural Activities. This component was intended to: (i) provide support for the modification of the Project Design based on lessons from the Creative Communities I project and findings from a related field assessment, (ii) develop cultural activities with PNPM teams during PNPM processes, such as the community forums, to collect inputs from poor and marginalized people, (iii) provide support for cultural activities through sub-grants to individual artists and community groups, (iv) provide support for pilot activities in three subdistricts to test the modified Project Design, and (v) provide support for the replication of the tested Project Design in other sub-districts. 3

17 Component II: Capacity Building. This component was designed to: (i) provide training of trainers on the use of cultural mediums to strengthen community empowerment, (ii) provide capacity building for cultural groups and civil society organizations in project and financial management, and (iii) provide technical assistance for PNPM Facilitators in using cultural approaches to enhance PNPM processes, including support for sharing best practices through video and online mediums. Component III: Monitoring and Evaluation. This component was designed to: (i) provide support for regular monitoring activities and the evaluation of Project effectiveness based on a monitoring and evaluation framework acceptable to the World Bank; and (ii) provide support for Project operations and the financing of an independent financial audit. 1.6 Revised Components The components remained unchanged during the project implementation. 1.7 Other Significant Changes CCII was a grant-financed operation, with resources made available through the PNPM Support Facility (PSF) multi donor Trust Fund. The idea was to provide an initial 6-8 month period and funding allocation for Yayasan Kelola (Kelola Foundation, generally known as Kelola), the implementing organization, to develop the concept and training materials and test the operations in a few locations (Phase 1). Lessons learned would then be incorporated into the revised program, and an additional allocation tailored to the revised design would enable them to prove the concept at a larger scale (50 locations) over a period of roughly 18 months (Phase 2), with the expected overall project implementation period spanning two years and ending in December However, the formal closing date was initially set at December 31, 2013 (just seven months) owing to an administrative need to coincide with PSF s formal closing date, an extension of which was then being negotiated. Once PSF was formally extended, CCII closing dates could be adjusted appropriately. Date November 29, 2013 June 13, 2014 August 22, 2014 Change Level II restructuring to extend the closing date to June 30, 2014, the anticipated end of Phase 1 activities. Immediately before the planned second restructuring for Phase 2, PSF underwent a transition process that saw the World Bank transfer support for a number of activities including CCII to a Managing Contractor (MC) funded by DFAT. A second Level II restructuring (World Bank, June 13, 2014) was completed to extend the closing date to September 30, 2014 to enable planning for a logical scale-down and handover of activities. With a delayed start to the DFAT MC, a third Level II restructuring (World Bank, August 22, 2014) extended the final closing date to March 31, 2015, and allocated an additional US$1.384 million. The decision was made to extend Phase 1 and implement activities in 24 locations in the interim period before DFAT commenced operations 4

18 December 29, 2014 (compared to the 50 locations planned under Phase 2), and the KPIs were adjusted. MoHA informed PNPM facilitators nation-wide that contracts would not be renewed and that PNPM program activities would halt effective December 31, No block grant was disbursed for Generasi was funded under a separate grant, and would continue as planned. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry Soundness of background analysis and lessons learned from previous projects. The design of Creative Communities II was built on the basis of proven success and experience from the previous grant for Creative Communities (CC I) (World Bank, November 16, 2007). Phase 1 of CC II focused on revising the design based on: a) key lessons documented from CCI (World Bank, June 2012); and, b) the results of a preparatory field assessment (Simatupang G. L., March 2013) that identified targeted opportunities for interventions. Lessons learned from CCI included that (i) cultural activities could effectively increase participation; (ii) participation generated by cultural activities was typically less formal and more inclusive; (iii) the divide between culture and development was difficult for communities, facilitators and cultural groups to grasp, and communities were typically confused by the open menu approach to cultural activities; (iv) the open menu approach resulted in demands for specialized support that were often outside the range of experience or skills of the implementing arts organization; and, (v) strong uptake of cultural activities to support development activities requires coordination between CCI and PNPM facilitators, and this requires support from MoHA/PMD. These lessons contributed to a theory of change on expanding participation in PNPM Rural, and were addressed in the design of CCII. 5 They were appropriate lessons, and the revised design was tested and evaluated at the end of Phase 1 to ensure it worked. Assessment of project design. The key changes in program design for CCII as a result of CCI experience were: focusing project objectives on participation; narrowing the range of activities used to achieve those objectives; and developing a clear facilitation strategy for communities moving from an experimental scattergun approach under CCI to building a program around a theory of change. This helped to eliminate the confusion communities had over preparing 4 With the passing of the Village Law (VL) in January 2014, policy decisions were made over the course of the year to plan for a winding down of PNPM, in an anticipated transition to VL implementation. Over most of 2014, stakeholders anticipated that the new government sworn in on October 20, 2014 would implement a final year of PNPM block grants in 2015 (the final year of the WB PNPM V loan) while preparations were completed for Village Law implementation. Contrary to expectations, the new government opted not to allocate PNPM block grants in the 2015 budget and shifted PNPM responsibility away from MoHA to the newly created Ministry of Villages, Disadvantaged Regions and Transmigration (MoV). MoHA officially conveyed these decisions to local governments on December 29, The theory of change described three pathways to achieving the project s PDO of active participation : 1) communities increase agency; 2) community representatives improve the quality of their representation of the marginalized and poor; and, 3) village facilitators and cultural workers increase capacity to facilitate development using creative approaches. See (Solvang, May 2015) page for more detail. 5

19 applications for open-menu grants, and enabled Kelola, the implementing organization, to provide more skilled, targeted facilitation to beneficiary villages. Instead of spending time recruiting diverse consultants to meet whatever demands communities put to them, they focused on tapping their networks for theater and video facilitators, and building the facilitators community development skills using a consistent framework and vocabulary. In order to maximize sustainability, the grants available to the communities to carry out the cultural activities were adequate to support basic activities, but small enough that the communities were encouraged to make material contributions, for example, providing clothing for theater costumes. Grants were kept small to avoid dependency and increase the potential for independent replication. The key performance indicators (KPIs) at the PDO level were tied directly to community participation in PNPM Rural IV 6, including (i) increased community participation in decisionmaking, (ii) increased participation of the poor and marginalized 7 in PNPM processes, and (iii) ability of PNPM facilitators and cultural groups to use cultural activities to support PNPM processes. These were reasonable indicators at the initial stages of the project when PNPM s parameters were clear and as captured in the PDO. The intermediate indicators had an appropriate focus on operational set up including technical guidelines and training materials during Phase 1 which were needed to support expansion during Phase 2. While increasing PNPM participation is an admirably clear and measurable objective, an alternative strategy for the PDO would have been to focus more broadly on empowering women, the poor and the marginalized to participate effectively in PNPM and village-level development activities. Framing it this way would have better reflected the projects real, underlying goals, and would have enabled a more nuanced evaluation of the CCII process and impact over time and a more natural transition into the village law framework. For example, an empowerment approach would have enabled the measurement of changes moving from attendance to participation, knowledge, motivation, access, influence and up to control. Applying this approach to both the development objective and the M&E strategy could have allowed for the development of indicators that were not primarily linked to PNPM decision meeting attendance. Since a number of those meetings were found to be organized on an invitation-only basis, CCII also faced an institutional challenge in those locations which it was not equipped to solve, in addition to the targeted social challenges it was intended to address. An empowerment approach would have provided additional flexibility to evaluate the project around such unanticipated constraints. Government commitment, stakeholder involvement, and participatory processes. The project was intended to support PNPM Rural implementation under MoHA/PMD. PMD committed to support the project, but at least for Phase 1 did not embrace it in the administrative ways they had agreed formally informing subnational facilitators of the project and encouraging them to support, cooperate with and/or integrate CCII activities into PNPM Rural processes. The strong commitment and investment by the implementing organization, Kelola, helped to overcome this gap at least at the local level, where they focused on developing strong working relations with the local PNPM Rural teams despite the (very important) lack of endorsement from the center. This situation improved somewhat for the Phase 1 extension when the PNPM Generasi Secretariat (at 6 Fourth National Program for Community Empowerment in Rural Areas (IBRD-80790) 7 In practice, marginalized was defined as no participation in PNPM and other village planning activities as the definition of marginalization in the six months prior to CCII. 6

20 central Government level) demonstrated clear support for the project. They participated in the field assessment as part of the preparations for Phase 1 Extension, and subsequently officially informed local Generasi facilitators and local governments of CCII activities and requested support and coordination. Consultations with the Generasi facilitators in the field yielded some effective collaboration, including developing strategies for targeting women and children s participation in West Java as part of its work on health 8. Risks and risk mitigation measures. The overall project risk was rated Moderate, which may have been somewhat underestimated owing to the absence at the time of project preparation of several key factors. The key risks identified in the Project Paper (PP) included stakeholder risk, implementing organization risk, and project risk. The risks linked to the project and to the implementing organization were appropriate, but the stakeholder risk would have benefited from more extensive analysis. The nature of the stakeholder risk was not detailed, but the mitigation measure noted that PSF would facilitate communication among stakeholders. Greater development of risks and mitigation measures (included in the proposal presented to and approved by the PSF Joint Management Committee (JMC) 9 ) could have been usefully included in the PP risk analysis: Risk of weak support from the central level of the GoI. Lack of clear information from PMD clarifying the purpose of CCII, its potential role in PNPM, and the roles and responsibilities of the facilitators could result in weak relationships and coordination at the field level. 2. Risk of a project mentality developing in the field that leads facilitators to view CCII and PNPM as separate programs. There was ample experience with this in other pilot activities (often with those implemented by PMD), and development of a strategy to avoid this could have been flagged for the initial stage of project implementation. An additional risk developed after the project began that could have been incorporated into the Restructuring Agreement completed in August In early 2014, the Dewan Perwakilan Rakyat (DPR) national parliament - agreed to end the utilization of loan funds for anti-poverty programs beginning in This led to the omission of PNPM block grants 11 from the draft Medium Term Development Plan prepared for the incoming administration, although stakeholders anticipated the new government 12 to exercise the mechanisms available to correct this in order to 8 The highest level of women s participation in CCII occurred in West Java (59%) largely due to its links to Generasi and its focus on maternal and child health. 9 (PNPM Support Facility, April 2012). The JMC proposal added that to address this risk, CCII will work closely with GoI counterparts in the preparation and pilot phase to identify an effective management structure and ensure that the roles and responsibilities of PNPM facilitators in relation to CCII are clearly conveyed to all PNPM personnel from the national down to the local levels. 10 Both of these additional risks were problems experienced in CCI and identified in the final GRM report, where there was confusion among PNPM facilitators about their relationship with the Creative Communities because of the project s lacking a formal relationship with PMD, and a project mentality that led to facilitators viewing CC and PNPM as separate programs. 11 While the PNPM block grants were funded wholly from the national and local government budgets, World Bank loans were used to fund facilitation, without which the grants could not be implemented. 12 The President was sworn in on October 20, 2016, and the new cabinet was announced on October 26,

21 provide a smooth transition to the Village Law in 2016 and not in This did not happen and resulted in the early end of PNPM field activities in December 2014, which left CCII without operational counterparts, and more importantly, without the central pillar of its PDO and corresponding KPIs (see sections 2.2 and 3.2 for further discussion). While this was far outside the anticipated risk framework during the initial project design, it could have been included or served as the basis for a more extensive restructuring process in August Completing the Phase 1 extension remained a sound objective as the new Village Law 14 was modeled on PNPM participatory processes, and the sub-grant activities during the extension were steered towards discussion of the Village Law and the PNPM-Village Law transition. Methods to strengthen and broaden community participation developed in CCII are equally useful under the new structures (see section 3.1). The project s Quality at Entry is rated Moderately Satisfactory due to the insufficient consideration of risk from a lack of GoI support, and a correspondingly weak mitigation strategy for stakeholder risk. 2.2 Implementation Factors outside the control of the implementing organization Supporting: Holiday timing of community presentations The planned phasing in of the 50 (original) communities in the Phase 2 scale up was originally developed so that activities would not overlap with the relevant community s religious holidays, given the intensive demands on individual time during those periods. For example, they planned to avoid working before and during Christmas in predominantly-christian Eastern Indonesia, and during the Muslim fasting month and holiday of Ramadan and Eid in Java. All project activities were however delayed for the three months that were required to restructure the remaining work and plan the transition to the DFAT Managing Contractor. The result was a modification of Phase 2 into a final Phase 1 Extension, with activities limited to a total of 24 locations: an additional eight locations from Phase 2 were added to the first 16 locations where training had already taken place, with all activities to be concluded by the new closing date of March 31, Kelola was forced to abandon the holiday work-around strategy in order to meet the advanced closing date: community performances and presentations were scheduled for the Christmas holidays in (Catholic) Flores, and during the Galungan-Kuningan holidays in (Hindu) Bali. In Indonesia, there is a traditional migration to home villages during these holidays, and as a result, the cultural groups enjoyed significantly larger-than-normal audiences looking for entertainment. All agreed it had a surprisingly positive impact and the result was better than the original plan of avoiding holiday overlap. Challenging: Lack of PMD support PSF was unable to develop an effective management relationship between Kelola and PMD in MoHA, and PMD issued no letters or memos to clarify the roles and responsibilities of PNPM facilitators with regards to CCII. This generally weakened the ability of CCII facilitators to push 13 As of December 31, 2014, only five of the 32 national regulations required to implement the law had been completed, and subnational governments had not yet been informed what local regulations would be required. 14 Undang-Undang Republik Indonesia Nomor 6 Tahun 2014 Tentang Desa 8

22 coordination with local PNPM processes. The exception to this was the involvement of and endorsement by the PMD unit in charged of Generasi during the Phase 1 Extension. Challenging: World Bank delays Yayasan Kelola, the implementing arts organization, responded to PSF s call for expression of interest in September 2012, and was selected by the panel 15 on November 14, Kelola started preparatory activities in January, with the first facilitator training in April in order to complete pilot activities before the Ramadan fasting month began in July. Yet the grant did not become effective until May 30, 2013 and the first disbursement was received from the Bank on July 31, 2013, which placed serious strains on the organization. Small organizations are generally poorly equipped to deal with the Bank s internal processes and delays. Challenging: PSF reorientation and transition During early 2014, the World Bank Office Jakarta, GoI and DFAT (PSF s principle donor), reoriented the role and direction of PSF. The result was a plan to transition a number of activities, including CCII, to a managing contractor to be mobilized by DFAT. Kelola s pilot had already been evaluated and they had hired and trained facilitators and local cadres for the first 16 villages in anticipation of continuation into Phase 2. Activities halted for three months while the project s future was renegotiated as part of PSF s restructuring, and the limited Phase 1 Extension covering 24 locations resulted. Challenging: PNPM s early closure The GoI decision to halt PNPM at the end of 2014 was not foreseen by the Bank, facilitators or the donor community (see the risk assessment in section 2.1 above for more detailed discussion). Up to late 2014, donors and much of GoI were confident that the incoming administration would revise the draft 2015 budget and re-fund the 2015 PNPM block grants, but this did not happen. As a result, the foundation for CCII s key objective and its KPIs no longer existed. Factors subject to the control of the implementing organization Supporting: Persistence in learning WB administrative processes The World Bank financial management (FM) and procurement processes are not simple, and the Bank does not have a training program for small implementing/recipient organizations. Kelola was very persistent in sending its administrative staff to PSF to sit with World Bank FM and Procurement staff to learn what they needed to know in order to meet Bank requirements, particularly with regards to managing and accounting for the sub-grants to local community groups and inexperienced NGOs. Supporting: Developing and implementing a subnational PNPM coordination strategy Kelola recognized early they had limited ability to leverage PMD and could not directly influence its lack of support for PNPM. In the absence of needed PMD support (see the relevant challenge in the section above), they developed a communications and collaboration strategy for subnational PNPM teams that succeeded despite the intensely top-down management of PNPM. They managed to persuade the South Sulawesi provincial PNPM Secretariat to issue a letter of support aimed at the districts and sub-districts, an achievement never seen in more than 15 years of (KDP and) 15 The review and selection panel consisted of four members, one each from PMD, PSF, BAPPENAS and AusAID (now DFAT, principal donor to the grant). 9

23 PNPM implementation. This helped them to coordinate activities at the village level in South Sulawesi. Challenging: Unclear M&E implementation As an arts organization learning about development, Kelola was largely unfamiliar with monitoring, evaluation and learning processes, and generally apprehensive about them. Kelola was happy for PSF to provide intensive technical guidance and implementation support for M&E, and to outsource to a number of consultants (see section 2.3 for more detail). As a result, they generally missed an opportunity to develop a more effective internal learning process. Midterm Review A midterm review was completed in January 2014, as planned, in preparation for Phase 2 implementation. This included an external evaluation of the Community Empowerment Process (CEP) (Gultom, December 2013). The review concluded that the pilot design was generally effective, and the project was making good progress towards its development objective. A series of case studies was developed for more detailed analysis of CEP and sub-grants experience (Widuri, August 2014). These were assessed and recommendations were made for moving forward (Sloman, September 2014) before the limited scale up in the Phase 1 extension. Effectiveness of risk mitigation. The mitigation of stakeholder risk as noted in the PP, with the WB team facilitating communication among stakeholders, was only weakly effective. PMD s management capacity was stretched by the national scale up of regular PNPM activities, and they focused on core responsibilities community participation levels (attendance at community meetings) were already meeting the KPI requirements in the PNPM Rural loan agreements, and thus CCII was not viewed as central to the program. Despite relationships with MoHA through the PNPM Rural loan and attempts to advocate for CCII, the WB team was generally unable to overcome this obstacle. As an ancillary project for which MoHA was not directly responsible, CCII easily slipped down PMD s list of priorities. With the exception of the detailed omitted risks identified in Section 2.1, the mitigation of other risks identified in the PP implementing organization risk and project risk was satisfactory. The mitigation of implementing organization risks is described in Section 2.4 below. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization M&E design. The project measured progress against the PDO using two closely related indicators linked to the communities participation in PNPM decision making meetings, and one to indicate the capacity of the PNPM facilitators and community groups to use the project tools. The August 2014 restructuring included some adjustments to the KPIs: it logically combined and clarified the two participation indicators, added an indicator on beneficiary understanding of the PNPM process, and retained the indicator on facilitator and community capacity. In the Project Paper, PNPM participation data would come from the PNPM M&E database facilitated by PSF. Kelola regularly gathered monitoring data on project activities, and this would be supplemented by a technical evaluation of the pilot, plus a baseline study implemented during the Phase 1 pilot, and an endline survey. Overall, the M&E design was straightforward and feasible, and the August

24 restructuring improved it by including participants understanding of the process, their actual participation in decision making, and facilitators abilities to use the tools developed. The units of measure for the PDO indicators were many (12) in order to capture any difference in impacts on women, youth, the marginalized and the poor. Its feasibility, however, changed in December 2014 when PNPM unexpectedly closed, there was no subsequent PNPM village planning meeting, the central element of the KPIs. M&E Implementation. Aside from data generated by its own activities, the project had limited data needs the KPIs required data on community participation in the annual planning meetings. The small number of pilot locations meant they could collect the data directly from PNPM facilitators without much difficulty. This changed in December 2014 when the PNPM facilitator contracts were ended and PNPM activities halted. With this final exception, Kelola was able to successfully monitor activities in the field through regular activity data collection and supervision missions, which supplemented the evaluations around Phase 1 activities. The baseline study 16 was conducted in November-December 2014, and the endline beneficiary perception survey 17 and qualitative case studies were completed in March M&E Utilization. The Bank and the implementing organization were able to utilize activity reports from the field together with findings from supervision missions and technical support missions to monitor, reconcile, verify and evaluate project implementation. Bank team activities were funded under a parallel Bank-Executed Trust Fund (BETF) designed to support CCII implementation. The overall M&E quality rating is rated Modest because of the lack of PNPM participation data in 2015 against which to calculate the KPIs. 2.4 Safeguard and Fiduciary Compliance Safeguards. Overall, safeguards compliance was rated Satisfactory. The project involved only community-based capacity building activities and did not involve any physical interventions, and no environmental or social safeguards were triggered. The Implementation Guidelines for Social and Environmental Safeguards (IGSES) 18, the safeguard mitigation framework under PNPM Rural was incorporated into the training modules. CCII training was conducted in a culturally appropriate and sensitive manner. The majority of activities incorporated local languages and sought to highlight local traditions in the project locations. Fiduciary & Governance. The project s fiduciary and governance compliance is rated as Satisfactory. Kelola hired additional finance staff who effectively managed soft expenditures and monitored sub-grant activities. Guidelines for sub-grant activities were developed and applied. Financial management practices met the Bank s requirements and enabled Kelola to provide accurate and timely financial reports. The Bank s final FM review noted (i) internal control systems at Kelola worked well; (ii) bookkeeping was managed properly with fully supported 16 (Pollock, March 2015) 17 (Migunani, 2015) 18 Updated to the Info Shop on April 1,

25 documentation; (iii) IFR and audit reports were submitted on time; and (iv) sub-grants were managed in a satisfactory manner. Procurement. The project s procurement compliance is rated Satisfactory. The Bank s Procurement Specialists noted that procurement has been carried out in accordance with the Bank s Procurement and Consultant Guidelines and in line with the updated procurement plan Kelola submitted for approval. 2.5 Post-completion Operation/Next Phase Transition arrangements. The project is being transitioned to a DFAT Managing Contractor now established with a focus on community and rural development. PSF, Kelola and GoI PMD and the Ministry of Villages, Disadvantaged Areas and Transmigration (MoV) 19 met as part of the ICR process to discuss lessons learned and future options. Any future activities will be managed from the Managing Contractor facility. Operations and Maintenance. The project did not involve any physical interventions, and no operations and maintenance is required. Follow-up projects. The DFAT Managing Contractor and GoI will consider options as the implementation of the Village Law progresses and opportunities develop to support inclusive participation. Future impact evaluation. Not applicable. 3. Assessment of Outcomes 3.1 Relevance of Objective, Design and Implementation Rating for Relevance of Objective, Design and Implementation: Substantial Relevance of Objective: Substantial The project was well-aligned with the Bank s CPS of , and remained aligned with the Bank s CPS 21, where it fits two key engagement areas: (i) Promoting Communities, Protecting the Vulnerable and Improving Health Outcomes, and (ii) Gender and Governance. At completion, the project was operating within a changed implementation PNPM structure under the GoI: PMD was required by parliament to cease the main PNPM operations one year ahead of schedule, and implementation of remaining grant-funded PNPM activities including Generasi were shifted to the MoV together with the launch of the Village Law. The CCII project remains highly relevant to the implementation of the Village Law, now the key GoI strategy for improving service delivery, strengthening institutions and enhancing the links of local government down to 19 The remaining PNPM close-down activities have been transferred from MoHA to the new MoV, which is responsible for implementing the Village Law. This models community participation processes directly on those established in PNPM. 20 (World Bank, July 22, 2008) presented to the Board on September 11, (World Bank, December 13, 2012) approved by the Board on December 11,

26 the village level. The Village Law directly incorporates the central CDD planning and decisionmaking mechanisms fostered by PNPM, and inclusive decision making remains key to the successful functioning of these processes. Relevance of Design and Implementation: Substantial The design and implementation of CCII remained relevant as they were built upon previous experiences with CCI using cultural activities that were demonstrated to increase the ability of the poor and marginalized to participate in PNPM decision making. 22 While PNPM is no longer the operational vehicle for decision making involving local development priorities and budgeting, the follow-on Village Law is built on the same principles and, based on the language contained in the law 23, is intended to employ the same village-level decision making mechanisms. The project used arts community theater and video in the main CEP, plus music, community TV, traditional performance, etc. in the sub-grants to create additional space for informal discussions on issues of critical concern to the community, from which they wrote and performed a public theater piece, created short videos or developed other creative expressions. Facilitators specifically targeted the poor and the marginalized for inclusion in activities, and created informal settings in which they would be more comfortable discussing issues with higher likelihood of participation. The products that emerged from this process were then discussed across the range of community stakeholders, and solutions developed that could be integrated into the planning process. The project aimed to provide an alternative mechanism for the poor and marginalized to participate in decision making processes where social and cultural norms would either preclude them from participating in formal village meetings entirely 24, or keep them silent if they did. The design attempted to address implementation problems at the lowest and most important levels of PNPM consultation and decision making, a problem which had deteriorated after the national scale up of PNPM and with weakened training and capacity building being provided to facilitators. The project attempted to develop a set of concrete tools to support the efforts under PNPM V to strengthen supervision and facilitation 25. There were several issues that emerged during project implementation, and which remain relevant to aspects of Village Law implementation. First, the design utilized the specialized skills of a CSO to work with subnational facilitators in the target locations to expand their ability to ensure inclusion of all members of a community in development processes, including the poor and marginalized, as part of a strategy to overcome PMD s overstretched capacity. PMD was straining to manage the national scale up, and had very limited ability to focus on improving the quality of PNPM program delivery when they were struggling with just delivering PNPM program activities. The CCII design did not fully anticipate PMD s concentrated focus on core PNPM deliverables and their ability to ignore all non-core work. The MoV a newly created institution has now been given responsibility for ensuring the same PNPM principles of participation, accountability 22 See (World Bank, June 2012) and (Simatupang L. G., 2012). 23 (Government of Indonesia, 2014) 24 A clear illustration of this is in Bali where PNPM and village planning follow customary law: all married men belong to the village committee, and women and youth are specifically excluded from participation. 25 See (World Bank, November 8, 2012), page 11 for more detailed discussion of the decline of facilitation processes. 13

27 and transparency are integrated into village management of development, and are similarly likely to find quality of implementation in 78,000 villages a challenge. The second issue that emerged was that in some locations the team found that community participation was limited by the practice of issuing invitations to participate in the PNPM decision making meeting. 26 If PNPM had not closed activities at the end of 2014, this could have constrained the project s ability to increase participation in decision making in these villages. Both the issues of the GoI s ability to support quality implementation and inclusive community participation in decision making will be very relevant to the successful implementation of the Village Law. The third issue, which will merit substantial consideration in any future CC activities, is a more developed advocacy strategy, particularly for the video outputs. The Theater for Development (TfD) activities incorporate a very strong community-based process in the theater formation process, and have a limited timeframe for performance outputs, by its nature performances take scarce time and resources. The Participatory Video activities also incorporate an inclusive consultation process, but the video outputs are easy and nearly costless to replicate and distribute on any scale. These two products lend themselves to very different advocacy strategies, and considerable effort should be applied to future designs for these. 3.2 Achievement of Project Development Objective To restate, the objective of the project was to improve the ability of villagers, especially poor and marginalized people, to participate actively in PNPM processes using cultural expressions in project locations. The project was implemented in two parts. Phase 1 was designed to prepare the manuals and training materials, test the revised design in 13 villages in three districts, and prepare administratively for a modest scale up. Phase 2 planned to cover implementation in a planned 50 sub-districts over 18 months. Sub-grant activities were intended to supplement the Creative Empowerment Process activities in the target sub-districts. Because of the PSF transition and handover of the project to the DFAT Managing Contractor (see Section 1.7), a limited Phase 1 Extension was substituted for Phase 2, and activities were undertaken in 24 villages, plus the subgrant activities in their related sub-districts, within a shortened six month timeframe. Rating: Modest Overview 1,261 people benefitted directly from the project as participants in 27 Creative Empowerment Process (CEP) groups, 54% of which were women. 26 This is a misinterpretation of a PNPM rule established a number of years ago to ensure broad community representation at PNPM decision meetings. The rule required at least 40 members of the community attend the decision meetings, with representatives of women s groups, the poor, traditional elders, and others, and some local officials issued invitations in order to ensure adequate representation. Over time, this practice has sometimes evolved into an understanding that only those with invitations may participate in the meetings. 14

28 CEP groups consulted with community members, prepared and presented 89 public theater and video performances seen by more than 12,300 people, including community members, PNPM and Generasi facilitators, village, sub-district and district government officials. (See Table 1.) 14 community-based organizations with 1,225 members (at least 48% female 27 ) received subgrants to utilize creative approaches to stimulate participation in village development processes, support Generasi health and education messages or raise awareness of the new Village Law, in 20 villages. (See Table 2.) Table 1: CEP Group Participants (Theater and Video) Facilitators KPK*s Villages Direct participants Total % Direct participants Female Indirect participants Cultural workers Project Phase Phase % 2,700 N/A Phase ,055 53% 28 9, Extension Total ,261 54% 12,300 N/A *Kader Pemberdayaan Kreatif (Creative Empowerment Cadre) Table 2: Sub-grant Recipients Project Phase Groups Total Group Members % Female Group Members Number of Presentations/ Discussions Audience Participants Phase % 10 ^ Phase *37% 10 3,650 Extension Total 14 1,225 *48% 20 3,650 * The gender was not identified for 77 group members, so this is a minimum percentage. ^ Data on audience participation was not collected for Phase 1. PDO indicators 50% increase in public participation in PNPM village planning processes in project locations, with 50% increase in participation by the poor, women and marginalized (respectively). While activities in 24 villages were ongoing during the Phase 1 Extension, MoHA announced the end of PNPM facilitator contracts in December 2014 and no sub-district block grants for As a result, there were no PNPM village planning processes completed in project locations and the indicator could not be calculated. However, the results from Phase 1 29 showed that attendance at PNPM planning processes increased by % when linked to creative processes. These 27 The gender was unspecified for 77 members, thus 48% is the minimum actual percentage of women. 28 The actual audience estimates were very conservatively reduced by half or more to compensate for individual attendance at more than one performance. 29 (Sloman, September 2014) page

29 increases in attendance were distributed robustly across the target groups, with a particularly strong response from the poor 30 : Poor: 268% Women: 93% At the same time, the end-of-extension perception survey noted that 70% of beneficiaries reported they were motivated by their experience to participate in formal village meetings, thus indicating that the participation PDO would likely have been met. Beneficiaries during the extension period included: Poor: 43% Women: 52% Youth: 56% Marginalized: 72% If it is assumed conservatively that the Phase 1 Extension result would have been marginally unsatisfactory in meeting this indicator while Phase 1 is considered fully satisfactory, and weight these ratings by the disbursement ratio, the following results are obtained: Table 3. Outcome weighted by disbursement Amount Weight Rating Outcome Phase 1 disbursements 606,000 39% Ph 1 Ext disbursements 964,000 61% Overall 3.77 Following standard rounding principles, this would have yielded a Moderately Satisfactory rating for anything better than or equivalent to a Moderately Unsatisfactory result from the Phase 1 Extension for this indicator. 70% of beneficiaries have increased understanding of PNPM and village planning processes, with 50% reported increased understanding by the poor, women, youth and the marginalized (respectively). 53% of all participants reported increased knowledge as a result of the project 31, with the following breakdowns: Poor: 45% Women: 55% Youth: 42% Marginalized: 40% This is somewhat below the target indicator for most of the target groups. The perception study reported consistent No results across the three provinces (15-20%), but moderated by I don t know responses possibly reflecting confusion about the ending of the PNPM program and uncertainty over the new processes under the Village Law. PNPM facilitators and cultural workers have increased capacity to use cultural mediums to improve participation in village planning, reflected in training modules and tools prepared, and >400 cultural workers implement activities to facilitate village planning. Four Creative 30 Marginalized were not identified in the Phase 1 evaluation data, and youth (age under 35) were added as a target group under the Phase 1 Extension based on Phase 1 results. 31 (Migunani, 2015). 16

30 Communities manuals were developed 32, and 31 CCII facilitators together with 24 village-level Creative Empowerment Cadres (KPKs) were trained in their use, which is the evidence available as a proxy to illustrate increased capacity. Roughly 1,300 local cultural workers implemented cultural activities linked to the CEP or Sub-grant processes aimed at facilitating village planning. This indicator is fully satisfied. The project achieved good results during the initial trial phase, but the full achievement of the PDO could not be measured because of the early closure of PNPM activities to which it was linked. End-of-project beneficiary perceptions survey data indicate the participation goal would possibly have been met, the facilitator capacity-building goals were exceeded, and the increase in knowledge was slightly below target. The advanced project closing date meant that the beneficiary perceptions survey and subgrant activities were simultaneous, and thus the survey did not include interviews with subgrant participants. Kelola was able to report basic participation numbers for the subgrants during Phase 1 Extension, but was unable to report on more detailed aspects of these activities. This means that the successes and the many challenges in the subgrants component are less visible. 3.3 Efficiency Rating: Substantial Efficiency weighs the project s benefits against the costs, and compares those with recognized norms to assess value for money. An empowerment project such as CCII would ideally measure the value of the increased benefit stream accruing to the poor and marginalized people in a PNPM community which integrates creative processes into its decision-making, or overall economic growth and the distribution of growth in the community, and compare those to a similar PNPM community which does not. Unfortunately, the CCII project timeframe together with the ending of PNPM block grants, did not permit such an analysis. Moreover, no similar analysis of a similar empowerment project (as compared to targeted economic empowerment) has been found in the literature 33. A second-best proxy is to compare the costs of CC II implementation with other empowerment programs. The first JSDF Grant to the Female Headed Family Empowerment Program (PEKKA) was considered a highly successful NGO-implemented project implemented during in Indonesia. Its objective was to empower widows (typically among the most marginalized poor in Indonesia) and initially had links to KDP, the predecessor program to PNPM. According to the Implementation Completion Memorandum 34, this start up empowerment program disbursed $2.112 million for 4,800 beneficiaries, or $440 per capita, over a period of approximately four years. The Financial Management team in PSF projects Kelola disbursed $1.57 million for 18,436 beneficiaries under CCII (see Annex 1), or roughly $85 per beneficiary. This comparison does not 32 One each for theater and video activities, a manual for effective facilitation using creative approaches, and one advanced module on how to manage the sustainable growth and continuation of a creative community, including planning, M&E and reporting. 33 The evaluation of economic empowerment has been recently surveyed in (Taylor, March 2014), and none were found to be fully satisfactory. In theory, economic empowerment should be easier to evaluate than general empowerment. 34 (World Bank, 2004) Trust Fund

31 intend to assess the nature or quality of participation, but is presented simply for purposes of rough comparison. Based on these results, CCII costs are considered reasonable. Table 4. CCII Participants (Phase 1 Extension) direct participants indirect participants total CEP activities 1,261 12,300 13,561 Sub-grants 1,225 3,650 4,875 Total 2,486 15,950 18, Justification of Overall Outcome Rating Rating: Moderately Satisfactory Relevance Achievement of PDO Efficiency Substantial Moderately Satisfactory Substantial The Objective, Design and Implementation remain very relevant to the implementation of the Village Law, in essence, the successor to PNPM. Project achievements started out very strong, and beneficiaries became much more pro-active as a result of the project. Definitive measurement of some results was not possible due to the unexpected cessation of PNPM activities in the communities, but weighting by disbursements yields Moderately Satisfactory 3.5 Overarching Themes, Other Outcomes and Impacts A basic cost comparison to a very successful, NGOimplemented women s empowerment program indicates it carries reasonable value for money. Poverty Impacts, Gender Aspects and Social Development. Ideally, greater voice and participation results in needs of the poor and marginalized being addressed by local government, and poverty reduction. However, such impacts require firm coordination with PNPM processes in order to establish a causal link, and would take far longer than one PNPM cycle to emerge. This CCII pilot was unable to demonstrate an impact on poverty, but this should be incorporated into future tracking activities. The project did effectively demonstrate its potential to establish safe spaces for women, the poor and marginalized to discuss issues and formulate input into village development, and to move away from silent acceptance of the status quo. A village chief was quoted in the Project Completion Report (PCR), With the activities of Creative Communities in Rianggede, our eyes have been opened and we understand better the aspirations of our community. The CCG members have 18

32 changed. Before they didn t voice their concerns, now they are brave enough to express what they want. 35 The beneficiary perception survey indicated some striking before-and-after differences in anticipated behavior. As shown in Figure 1 below, more than half said that before the project they would have done nothing in response to a problem or complaint in the village, and fewer than 10% would have brought an issue to a community meeting. In contrast, after their experience with CC II, just 6% would ignore a problem or complaint, and more than 30% say they would take it to a community meeting for discussion. In addition, while less than the target 70% reported increased understanding of development processes, a full 70% of beneficiaries said they were more motivated to participate actively in community meetings. Given these significant changes, it is somewhat paradoxical that only 53% of participants reported increased knowledge as a result of the project. Nonetheless, the CEP clearly created space for public discussion of local concerns that did not exist before, and people felt more empowered to pursue solutions. Figure 1: Community members perceptions about what they would do when there was a problem or complaint in the village before and after CC II Percentage Do nothing Discuss with friend/neighbo rs Bring to community meeting Other Don't know Before CC After CC Institutional Change. The key question here is whether Creative Communities has the power to change sub-district and village-level meeting behavior where participation and decision making is limited to 40 invited persons. This rule of 40 was created as a minimum number of participants needed in decision-making meetings, but has been commonly reinterpreted as the maximum required. The result is local decision making with limited community participation and characterized by elite capture. Phase 1 results indicated a % increase in people attending PNPM decision making meetings when they were linked to CC II activities, with strong increases by demographic groups (poor, marginalized and youth) not typically represented in these meetings. The absence of PNPM meetings during or following the Phase 1 Extension means these results could not be tracked any further. 35 Dewa Putu Arya, Head of Rianggede Village in Bali. (Solvang, May 2015) page

33 3.6 Summary of Findings of the Beneficiary Perception Survey and Stakeholder Workshops Beneficiary perception survey. As the project closed, Kelola (with technical inputs from PSF) commissioned a beneficiary perception survey, including views of village chiefs, PNPM facilitators, KPKs, and members of creative community groups. The key findings of this study are summarized in Annex 3. Stakeholder workshop. On 17 June, 2015, PSF invited key stakeholders to a workshop in Jakarta to comment on the initial findings of the project evaluation presented in this ICR. The summary of the discussion amongst participants is included in Annex Assessment of Risk to Development Outcome Rating: Moderate The technical risk is rated moderate. A key question on the sustainability of the development outcome is the communities technical ability to continue or replicate cultural processes to support inclusive planning and decision making. More than 90% of both PNPM facilitators 36 and CC II KPKs reported in the Perceptions Survey that they felt CC II had an impact on the PNPM planning process in the villages 37, and 94% were optimistic that the CC approach would be adopted as part of future planning processes. 79% of the facilitators were confident in their ability to implement the CC approach, including 96% of the video facilitators. Nonetheless, the implementation of video activities in particular will depend on continued access to equipment. The political risk is rated as moderate. The large number of indirect beneficiaries in the CC II project in the form of (initially) consultation groups and (subsequently) audiences meant that it was a popular program with potentially large impact, but also means that it could be viewed as either a threat to local elites and office holders, or vulnerable to capture. There was one example in NTT where the village chief was initially a supporter but soon perceived the activity as potentially undermining his authority and withdrew from active support. In another village, the strong support and participation of the village chief in CC II activities caused the CC team to self-censor their video subjects in order to avoid conflict. Facilitators will have to build their skills to avoid the creative tools becoming divisive instead of inclusive. The Government ownership risk is rated significant. Faced with the daunting task of implementing the Village Law in more than 70,000 villages across Indonesia, the GoI is highly unlikely to obstruct CC activities but at the same time it is unlikely to be able to organize active support for this approach to generating more inclusive participation. To the extent local governments are able to obtain experience and training in the CC approach, they are likely to accept the process and to support it based on results and popular community response 63% of the village chiefs included in the perception survey felt their limited experience had improved the village or positively changed villagers mindset. 36 As of May, 2015, the 13,000 ex-pnpm facilitators were being re-hired by the MoV as village facilitators to assist with planning and decision making under the Village Law implementation. 37 There were three (?)villages in Bali with CC II activities that did not receive PNPM block grants, but these were apparently not identified by the perceptions survey team. It is unlikely facilitators would think the project contributed to the PNPM planning process in these locations. 20

34 Based on the above, the overall risk to development outcome is rated moderate. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory The design of the CC II project was based on previous experience with CC I and the evolving challenges the GoI was experiencing in maintaining quality facilitation and participation as PNPM scaled up to national coverage. The Bank worked together with the GoI and the Grant Recipient to incorporate lessons learned from CC I, most notably the need to narrow the scope of creative activities and to create standardized operations and training manuals essential for replication. The only significant weakness in the design or risk matrix was underestimating the concentrated emphasis PMD would place on what it viewed as its core mission (PNPM and Generasi) and its limited willingness to divert attention from that. PNPM had a highly centralized management structure, and subnational teams were accountable to the top they were generally unwilling to innovate without specific instructions, especially if it meant extra activities on top of an already heavy workload. The result was that while PNPM field facilitators were interested in CC II processes, they were reluctant to collaborate directly without clear instructions from PMD, and Kelola had to apply additional resources to overcome this. Based on the above, the Bank s quality at entry is rated moderately satisfactory. (a) Quality of Supervision 38 Rating: Moderately Satisfactory Three issues were identified which affected project implementation. There were substantial initial delays in contracting and disbursement, during which Kelola prefinanced project activities a challenge for a small arts foundation. They were selected by an evaluation panel on November 14, 2012, the project was approved in late May 2013, and disbursement occurred on July 31, In order to meet the PNPM planning cycle and the planned project schedule, Kelola had to complete the first Creative Empowerment Process in the pilot areas before the Ramadan fasting month began on July 9, 2013, which meant facilitator training had to take place in April 2013 with operations manuals and training modules ready. Kelola pre-financed approximately six months of project activities. The reorganization of PSF activities during described in Section 1.7 resulted in a three month delay in project activities while the project curtailment and transition to the DFAT managing contractor were planned. This created substantial uncertainty and job insecurity for 38 ICR Guidelines: the extent to which the Bank proactively identified and resolved threats to the achievement of relevant development outcomes 21

35 project staff and facilitators who had been trained and were ready to move into communities for the implementation period The GoI environment during the last year of the project was substantially different to that encountered during project design: the national parliament prohibited the use of borrowed funds to finance anti-poverty projects after 2014, which had clear implications for PNPM activities. There was thus a known risk that PNPM would cease activities at the end of the year this did happen, and meant there was no way to effectively measure achievement of the CC II s PDO that was explicitly tied to PNPM participation. An alternative strategy would have been to acknowledge this risk during the August 2014 restructuring, possibly modify the PDO to refocus on the importance of inclusion and empowerment in the PNPM transition to Village Law, and adapt the perception survey accordingly. Based on these factors, the Bank s quality of supervision was rated moderately satisfactory. (b) Justification of Rating for Overall Bank Performance Rating: Quality at Entry Moderately Satisfactory Quality of Supervision Moderately Satisfactory In light of the ratings of the elements noted above, the Bank s overall performance is rated Moderately Satisfactory. 5.2 Recipient Performance Rating: Satisfactory The grant recipient, Yayasan Kelola, demonstrated strong overall commitment to the project from beginning to end, including pre-financing six months of activities in order to meet the overall project schedule. They consulted regularly with the Bank and organized field visits. They hired skilled operations staff, and added staff and skills for financial management and procurement in order to meet Bank requirements. They established good working relationships with PNPM counterparts in the field, despite the absence of directives from PMD. In South Sulawesi, this resulted in a provincial instruction to facilitators to coordinate with and support CC II activities, the only time the Bank recalls this happening in KDP/PNPM. 6. Lessons Learned The lessons learned from the CC II experience fall under three main categories: a) the ability of creative approaches to engage the marginalized, poor women and youth; b) the impact of creative approaches on individuals in the community; and c) the ability of creative approaches to affect the planning process. 22

36 The ability of creative approaches to engage the marginalized, poor women and youth CC II has demonstrated that creative approaches are highly effective in engaging the marginal, poor, women and youth. While 87% of the facilitators and most of the community cadres (KPKs) were male 39 (typically an obstacle to women s participation) more than half of direct participants in CEP activities were female, indicating the approach is well-suited to women s participation. Moreover, during the Phase 1 Extension, 43% of participants were classified as poor, 72% were marginalized, and 36% were youth under 17. Youth were not initially targeted in Phase 1, but results showed that they were often excluded from village planning processes, and at the same time were attracted by the opportunity to learn about new technology (video), and were successfully incorporated in the Extension. The impact of creative approaches on individuals in the community CC II demonstrated that creative approaches were effective in increasing the agency of CCG members, defined as an increase in self-esteem, ability to express themselves, strengthened networks, a stage for expression and gaining understanding of participation as a right. Figure 2 shows beneficiaries perceptions about the individual benefits of participating in the project (Migunani, 2015). Figure 2. Beneficiaries of CCII CEP in Phase 1 Extension perceptions of what they gained from participating in Creative Communities (in response to open ended questions allowing multiple answers, in percentage) More activities in the village, 18.5 Added organizational experience, Get to know other communities in other villages, 7.82 More aware of my rights to participate in village development, Can access important people such as KPK, KPMD, village gvmnt, 8.12 Gained place and friends to disuss with and express opinions, Learn new ways of expressing complaints, More self confident, Learned new skills/knowledge 38.2 Understand situation in village better, There were no significant differences in how the poor and non-poor perceived these benefits, indicating that the approaches are equally effective in empowering the poor as the non-poor. 39 This was despite the concerted efforts of Kelola to enlist women in these roles. 23

37 Creative Empowerment Processes result in participants becoming more pro-active in the community. Figure 1 (see section 3.5) shows what the beneficiaries perceive they would do in response to complaints and problems in the community both before and after the project: 53% reported they did nothing in response before the project, which fell to just 6% after the project. Beneficiaries also perceived themselves to be generally more active in the community after participating in the project, and 70% of beneficiaries in the CEP Phase 1 Extension reported to be more motivated to participate in formal meetings in the villages after the project. The ability of creative approaches to affect the planning process. Phase 1 provided evidence that when creative activities are integrated into planning processes, creative approaches are highly effective in increasing attendance at formal meetings motivating people to participate, and in bringing the voices of the marginalized and poor into decision making processes. This requires joint planning by stakeholders, and an adequate timeframe for implementing activities. Creative approaches affect participation in various ways, and it is important to define the project objectives clearly and select indicators which capture the underlying PDOs. The CCII PDO included active participation which translated into increased attendance at PNPM meetings in the KPIs, and did not capture the real impact of the project on community empowerment. Timely assessments and studies can provide critical information about the barriers and opportunities for participation and empowerment. For example, the baseline study highlighted larger opportunities to impact participation at the sub-village level or to improve the quality of representation rather than focus on mass attendance, but it emerged too late to effect the process or the KPIs. Creative approaches in formal community planning processes generates community interest and help to break down formal power structures that stand as barriers to the participation of the marginalized and the poor. CC II Phase 1 showed examples of how attendance at PNPM events increased as a direct result of the project. By offering a service to village government in attracting community members to formal meetings, creative communities could generate support for village government and increase leverage and opportunities for the marginal to voice their concerns. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Donors The implementing agency, GoI counterparts from BAPPENAS, MoHA, and MoV provided comments during the Stakeholder Workshop. See Annex 6 for more details. Their inputs were consistent with the findings presented in this ICR. 24

38 Annex 1. Project Costs and Financing (a) Project Cost by Component (in US$ million equivalent) Components Original Allocation (USD millions)* Actual/Latest Estimate (USD millions)** Percentage of Revised Total 1. Community-based cultural activities % 2. Capacity building % 3. Monitoring and evaluation % Total Baseline Cost % Physical Contingencies Price Contingencies Total Project Costs % * The table shows only the breakdown of component executed through RETF. Refer to Project Paper approved by Sector Manager on May 21, 2013, the project cost was USD4.7mil to cover activities implementation for 2.5 years for Phase 1 and 2. ** The original grant amount to the recipient was USD606K for phase 1, and amended to USD1.92mil on September 26, As PSF has gone through a transition process, Creative Communities II is one of the projects that will transition out of the Bank in 2015 to be managed by Managing Contractor funded by DFAT. (b) Financing Source of Funds Type of Cofinancing Appraisal Actual/Latest Estimate Estimate (USD millions) (USD millions) Trust Funds Indonesia - Program for Community Empowerment % Percentage of Appraisal 25

39 Annex 2. Outputs by Component Component I: Community-based Cultural Activities In Phase 1, more than 1,200 individuals (participating villagers, PNPM facilitators and cultural workers) participated directly in CCII activities, while more than 5,600 participated indirectly (villagers and others who attended presentations and/or performances). The project provided support for 14 types of cultural activities, which resulted in a total of 37 presentations and/or performances in 13 villages in seven sub-districts. Each presentation and/or performance was jointly organized with, or attended by, PNPM Rural and/or PNPM Generasi personnel. The majority of the participants were women, children and youths who had little or no previous knowledge of PNPM Rural or PNPM Generasi. Table A2.1: Locations and types of activities funded in Phase I Province District Sub-district Village Modality West Nusa Tenggara North Pemenang East Pemenang, CEP: Video Lombok Malaka and Nipah Hamlet South Sulawesi Bone East Tanete Riattang Bajo e CEP: TfD West Kalimantan Landak Sengah Temila Sahamp CEP: TfD & Video West Kalimantan Landak Sengah Temila Petai CEP: TfD & Video West Nusa Tenggara North Lombok Bayan Sukadana Sub-grants West Nusa Tenggara North Lombok Pemenang East Pemenang Sub-grants South Sulawesi Bone East Tanete Riattang Bajo e Sub-grants South Sulawesi Bone Bontocani Langi Sub-grants South Sulawesi Bone Bontocani Kahu Sub-grants South Sulawesi Bone Bontocani Patukku Sub-grants West Kalimantan Landak Sengah Temila Sebatih Sub-grants West Kalimantan Landak Sengah Temila Gombang Sub-grants West Kalimantan Landak Sengah Temila Maro o Hamlet, Gombang Sub-grants West Kalimantan Landak Jelimpo Pawis Hilir Sub-grants 3 Provinces 3 Districts 6 Sub-districts 11 Villages Table A2.2: Locations and Types of activities funded in Phase I Extension Province District Sub-district Village Modality Bali Jembrana Negara Berambang CEP: Tfd Bali Buleleng Sukasada Panji CEP: Tfd Bali Bangli Bangli Pengotan CEP: Tfd Bali Badung Abiansemal Desa Bongkasa Pertiwi CEP: Tfd Bali Karangasem Bebandem Budakeling CEP: Tfd Bali Karangasem Jasri CEP: Tfd 26

40 Province District Sub-district Village Modality Bali Gianyar Tampaksiring Manukaya CEP: Tfd Bali Tabanan Penebel Riang Gede CEP: Tfd Flores - East Nusa Tenggara Manggarai Cibal Barang CEP: Video Flores - East Nusa Tenggara Wae Rii Golo Wua CEP: Video Flores - East Nusa Tenggara Manggarai Timur Borong Waling CEP: Video Flores - East Nusa Tenggara Poco Ranaka Lento CEP: Video Flores - East Nusa Tenggara Sambi Rampas Golo Wangkung CEP: Video Flores - East Nusa Tenggara Ngada Golewa Malanuza CEP: Video Flores - East Nusa Tenggara Soa Ngabeo CEP: Video Flores - East Nusa Tenggara Inerie Manubhara CEP: Video West Java Sukabumi Caringin Caringin Wetan CEP: Tfd West Java Garut Bayongbong Panembong CEP: Video West Java Bandung Barat Cililin Cililin CEP: Video West Java Cianjur Cikalong Kulon Sukamulya CEP: Tfd West Java Kuningan Cilimus Kali Aren CEP: Video West Java Kuningan Garawangi Garawangi CEP: Video West Java Subang Tanjung Siang Cimeumal CEP: Tfd West Java Sumedang Ganeas Cikoneng CEP: Tfd Bali Bangli Bangli Sidem Bunut Sub-Grant Bali Gianyar Tampaksiring Tampaksiring Sub-Grant Bali Gianyar Tampaksiring Sanding Sub-Grant Flores - East Nusa Tenggara Ngada Golewa Malanuza Sub-Grant Flores - East Nusa Tenggara Ngada Golewa Ratogesa Sub-Grant Flores - East Nusa Tenggara Ngada Soa Mengeruda Sub-Grant Flores - East Nusa Tenggara Ngada Soa Piga I Sub-Grant Flores - East Nusa Tenggara Ngada Soa Bogoboa Sub-Grant West Java Kuningan Cilimus Bandorasa Wetan Sub-Grant West Java Subang Tanjung Siang Cikawung Sub-Grant 3 Provinces 17 Districts 24 Sub-districts 33 Villages Component II: Capacity Building Capacity building activities included the drafting and revision of project manuals for theater for development and community video production, a manual for effective facilitation using creative 27

41 approaches, and a module on how to manage the sustainable growth and continuation of a creative community, focusing on managerial aspects such as planning, M&E and reporting. Training in facilitation skills was provided to 31 CC II facilitators and 24 creative community cadres, who subsequently trained 1261 members of creative community groups. Training was also provided to 12 sub-grantees in project development, implementation, and reporting. Component III: Monitoring & Evaluation Regular field monitoring was completed effectively, and reports were submitted to PSF in accordance with the schedule. Analytical products included an initial field assessment on opportunities (Simatupang G. L., March 2013); a Phase 1 evaluation of the theater and video creative processes (Gultom, December 2013); a summary evaluation of Phase 1 including the subgrants and planning for Phase 2 (Sloman, September 2014); the baseline study (Pollock, March 2015); and the end-of-project beneficiary perceptions survey (Migunani, 2015). 28

42 Annex 3. Economic and Technical Analysis As stated in Section 3.3, efficiency weighs the project s benefits against the costs, and compares those with recognized norms to assess value for money. An empowerment project such as CCII would ideally measure the value of the increased benefit stream accruing to the poor and marginalized people in a PNPM community which integrates creative processes into its decisionmaking, or overall economic growth and the distribution of growth in the community, and compare those to a similar PNPM community which does not. Unfortunately, the CCII project timeframe together with the ending of PNPM block grants did not permit such an analysis to be undertaken. Moreover, no similar analysis of a similar empowerment project (as compared to targeted economic empowerment) has been found in the literature 40. Since this wasn t feasible, a second-best proxy was used to compare the costs of CC II implementation with other empowerment programs. The first JSDF Grant to the Female-Headed Family Empowerment Project (PEKKA) was considered a highly successful NGO-implemented project implemented during in Indonesia. Its objective was to empower widows (typically among the most marginalized poor in Indonesia) and initially had links to KDP, the predecessor program to PNPM. According to the Implementation Completion Memorandum 41, this start up empowerment program disbursed $2.112 million for 4,800 beneficiaries, or $440 per capita, over a period of approximately four years. The Financial Management team in PSF projects Kelola disbursed $1.57 million for 18,436 beneficiaries under CCII (see Annex 1), or roughly $85 per beneficiary. This comparison does not intend to assess the nature or quality of participation, but is presented simply for purposes of rough comparison. Based on these results, CCII costs are considered reasonable. From a technical perspective, the project team constructed a Theory of Change which was reflected in the key project documents, including the project proposal, facilitators manuals, and training materials. The theory of change highlighted several pathways towards achieving the project PDO of active participation, where active participation means: Awareness about PNPM and/or other development processes Motivation to participate Access to participate Ability to influence decision making in the development The pathways for achieving active participation included: 1) communities increase agency, including improved understanding of their right to participate, self-esteem, capacity for selfexpression social networks and areas for expression, and understanding of PNPM and development processes; 2) community representatives improve quality of their representation of the marginal and poor; and, 3) facilitators and cultural workers increase capacity to facilitate development using creative approaches. 40 The evaluation of economic empowerment has been recently surveyed in (Taylor, March 2014), and none were found to be fully satisfactory. In theory, economic empowerment should be easier to evaluate than general empowerment. 41 (World Bank, 2004) Trust Fund

43 The objective was to employ creative approaches (that is, participatory video, theater for development and grants for creative proposals) to empower community members, specifically the poor and marginalized, to increase their capacity to participate actively in PNPM and development processes. The technical success of the project was reflected in the changes and results documented in the Phase 1 evaluation (Gultom, December 2013), and in the perceptions survey conducted at the end of the Phase 1 Extension (Migunani, 2015) (see Annex 5 for more detailed information on the perceptions survey results). Based on results of the Phase 1 evaluation, CCII s creative initiatives were effective in increasing people s understanding of PNPM and in creating opportunities to participate in PNPM. They were also more likely to participate in PNPM forums and be involved in PNPM in general following this involvement. Attendance of the marginalized, poor, women and youth in PNPM forums increased, participants at such forums included demographic groups whose members rarely or never previously attended such forums. The Phase 1 evaluation also documented examples of increased quality of participation by women and poor members of the community in village planning processes. The project was unable to document increased participation as a result of activities in the Phase 1 Extension owing to the unexpected fact that PNPM did not continue activities into 2015, hence no planning meetings were held. The Phase 1 evaluation findings showed that the project facilitated some changes of agency, empowerment and capacity of villagers, including the following: Improved self-confidence, particularly in terms of voicing views and issues related to community level development initiatives; Ability to undertake discussions, including expressing one s views and valuing the views of others; Ability to work as a team; Increased sense of community; Increased technical capacities in terms of the use of video, theater and other forms of creative expression. These results were supported by results of the Phase 1 Extension perceptions survey, where people reported increased self-confidence, increased ability to express complaints, increased understanding of the situation in the village and increased awareness of their rights to participate in development processes. Strengthened networks were also highlighted by some respondents, such as the opportunity to access important people in the village, having a place to meet and discuss with friends and a chance to meet with communities in other villages. Perhaps the most striking indicator of increased agency is the comparison of participants perceptions about what they would do when there was a problem or complaint in the village before and after CC I, summarized in Figure 3.1 below (Figure 7 from the Project Completion Report (Solvang, May 2015)), shown below. The results show that before the project, more than half the participants would do nothing in response to a problem or complaint in the village, while after participating in CCII, this number dropped to only 6%. 30

44 Figure 3.1: Participants perceptions about what they would do when there was a problem or complaint in the village before and after CCII Percentage Do nothing Discuss with friend/neighb ors Bring to community meeting Other Don't know Before CC After CC Overall, the results of the two evaluations indicate that the empowerment techniques used in the project were technically successful in helping communities expand participation and address issues in the village. 31

45 Annex 4. Loan Preparation and Implementation Support/Supervision Processes (a) Task Team members Names Title Unit Responsibility/ Specialty Lending/Grant Preparation Jan Weetjens Sector Manager EASID Sector Manager Susanne Holste Lead Social Development Specialist EASID Task Team Leader Kevin A Tomlinson Senior Operations Officer EASID Portfolio Management Kok Eng Amy Sim Social Development Specialist EASID Project Coordinator I Gusti Ngurah Wijayakusuma FM Analyst EASFM Financial Management Unggul Suprayitno Senior Financial Management Specialist EASFM Financial Management Hanggar Irawan Operations Analyst EASID Financial Management Yogana Prasta Operations Advisor EACIF Operations Mariangeles Sabella Senior Counsel LEGES Legal Ria Nuri Dharmawan Associate Counsel LEGES Legal Ahsan Ali Lead Procurement Specialist EAPPR Procurement Achmad Zacky Wasaraka Procurement Analyst EAPPR Procurement Zulfi Novriandi Procurement Analyst EASID Procurement Inge Sutardi Tan Program Assistant EASID Administration Sentot Surya Satria Social Development Specialist EASID Task Team Leader Imelda Luciana Noto Trust Fund Coordinator EACIF Trust Fund Peter Leonard Regional Safeguards Adviser EASOS Safeguards Juan Martinez Senior Social Scientist EASIS Safeguards Dennie S. Mamonto Consultant EASIS Safeguards Supervision/ICR Natasha Hayward Senior Social Development Specialist GSURR Task Team Leader Caroline Mary Sage Senior Social Development Specialist GSURR Task Team Leader Kok Eng Amy Sim Consultant GSURR Operations Inge Sutardi Tan Program Assistant EASID Administration Jacqueline Pomeroy Consultant GSURR Operations Kevin A Tomlinson Acting Practice Manager GSURR Practice Manager Narae Choi Young Professional YPP Portfolio Management Niruban Balachandran Operations Officer GSURR Portfolio Management Juan Martinez Senior Social Scientist GSURR Safeguards Achmad Zacky Wasaraka Procurement Analyst GGODR Procurement Ahsan Ali Lead Procurement Specialist GGODR Procurement I Gusti Ngurah Wijaya Kusuma FM Analyst GGODR Financial Management Hanggar Irawan Operations Analyst GSURR Financial Management Festina Lavida Operations Officer GSURR Financial Management Unggul Suprayitno Senior Financial Management Specialist GGODR Financial Management Yogana Prasta Operations Advisor EACIF Operations Imelda Luciana Noto TF Coordinator EACIF Trust Fund Jana Halida Uno Operations Officer EACIF Operations Mariangeles Sabella Senior Counsel LEGES Legal Ria Nuri Dharmawan Associate Counsel EACIF Legal Kian Siong Environmental Specialist GENDR Safeguards Ani Himawati Operations Analyst GSURR Operations/Safeguards 32

46 (b) Staff Time and Cost Staff Time and Cost Stage of Project Cycle USD Thousands (including travel and No. of staff weeks consultant costs) Lending Total: Supervision/ICR FY FY FY Total: * Staff time and cost was financed through BETF/TF

47 Annex 5. Beneficiary Perceptions Survey A beneficiary perceptions survey (Migunani, 2015) was conducted after activities were completed in each of the three provinces included in the Phase 1 Extension. Twenty-four project villages were targeted, with structured beneficiary samples in each village: Beneficiary Target per Village Total Resulting Sample Village Chief 1 24 (24 achieved) PNPM Facilitators 2 48 (48 achieved) Creative Empowerment Cadres, or KPK 1 24 (24 achieved) Creative Community Member (665 achieved) Some of the key question areas and results from the survey are summarized here. Was the Creative Empowerment approach useful? What did group members gain from the experience? Will they participate in future planning activities? Have behaviors changed? The village chiefs thought the approach was generally useful, and 23 out of 24 expected to continue to utilize what they learned from the pilot. 22 out of 24 village chiefs had followed up on the issues raised as part of the pilot 38% felt the activities had helped them to understand the issues 33% felt the activities and results accurately reflected the people s aspirations PNPM facilitators had similar views on the usefulness of the Creative Empowerment approach. 92% though the project had a positive impact on PNPM planning processes, with the community video (96%) thought to have a slightly higher impact than theater activities (89%). 90% of facilitators followed up on issues developed during CCII activities Facilitator engagement in CCII activities was strongly correlated with their willingness to follow up a critical issue for future planning and design. Even the local KPKs felt (23%) they had a better understanding of village problems with CCII. More importantly, 88% said they felt CCII activities had raised awareness of others, brought the communities closer together, and helped people become more open in expressing their views. Individual members of the video and theater groups highlighted the value of strengthened networks, such as the opportunity to access important people in the village, having a place to meet and discuss with friends and a chance to meet with communities in other villages. 34

48 Figure 5.1: Perceptions about individual benefits gained from being members of CCGs (in percentage) 42 Can access important people (i.e. KPK, More aware of my rights to participate Get to know other communities in other More activities in the village Learned new skills/knowledge Gained place and friends to disuss with Added organizational experience % of non-poor % of poor % of total Understand situation in village better Learn new ways of expressing complaints More self confident Participants perceptions about their knowledge of PNPM and village planning process were more mixed. An average of 53% said they increased their knowledge of PNPM and village planning processes, with a large percentage of I don t know responses, possibly because of the early ending of PNPM activities and uncertainty about the implementation process for the new village law. Figure 5.2: CCG core members' perceptions about whether their knowledge about PNPM and other village planning and development processes increased Yes No Don t know 42 Respondents were asked an open ended question about the benefits of participating in CCGs. They were allowed multiple answers, which means the total of responses exceed 100%. 35

49 One of the most significant findings of the perceptions survey was that CCG members perceptions about themselves as active participants in the community have changed over time with their participation in the project. Figure 5.3 below shows what core CCG members perceive they did when there were problems or complaints in the community before the CCII and what they perceive they do now after the project. Figure 5.3: CCG core members' perceptions about what they would do when there was a problem or complaint in the village before and after CCII Percentage Do nothing Discuss with friend/neighb ors Bring to community meeting Other Don't know Before CC After CC The figure shows that 53% said that they used to do nothing to respond to complaints or problems before the project. This is in sharp contrast to 6% who said that after the project they would do nothing. People see themselves as more pro-active now after participating in CCGs, and they will bring complaints and problems into discussions with friends and neighbors, into community meetings or find other ways to address the problems. Moreover, nearly 70% of CCG members said they are now more motivated to participate actively in community meetings compared to 6 months ago (Figure X). In NTT and West Java around 80% reported they were now more motivated than before. Only 46% of the respondents in Bali said the same, and more people in Bali stated they did not know, likely due to the inactivity of PNPM in Bali where PNPM Rural closed its operations the previous year, which naturally caused challenges in creating links between CCII, PNPM and other development processes. Figure 1.4: CCG core members' perceptions about their motivation to participate in village meetings compared with six months earlier Bali West Java NTT Total Yes No Don't know 36

50 Annex 6. Stakeholder Workshop Report and Results Date: 17 June 2015 Location: PNPM Support Facility Participants: See attendance list following Agenda 1. Welcome remarks and objectives of meeting - Natasha Hayward, Creative Communities II Task Team Leader (TTL) 2. Participants introduction 3. Presentation of results and lessons of Creative Communities II (CC II) - Jackie Pomeroy, Writer of CC II Implementation Completion Report (ICR) (See Annex 2 for Presentation Slides) 4. Questions and feedback from participants 5. Summary and closing remarks - Natasha Hayward Summary of Discussion 1. Objective of Meeting The objective is to seek feedback for the CC II ICR and to discuss, more broadly, the project s values and lessons for future replication. Natasha explained that she is relatively new to the project having taken over the TTLship from Caroline Sage in January Questions and feedback from participants a) Creative Approaches are valuable for community driven development There is a general consensus that the use of creative approaches in CC II has value in increasing the diversity of voices in community planning, and despite the closing of PNPM, remains very relevant in the new Village Law system. Creative approaches are important for overcoming the culture of silence and counteracting the effects of elite capture in community driven development. This is especially relevant now that the principle of participatory planning has been enshrined in the Village Law. CC II motivates the community to identify collective interests and take collective actions. (Bito, MOV). Community television and participatory video, for example, are particularly useful for communication and outreach to the community (Harno, KMN). CC II is a small example that provides important lessons on how cultural activities can break down formal barriers between the government and the community, and provide informal spaces for village planning facilitators and the community to do things differently (Prahastanto, WB). b) PDO Rating Jackie seeks inputs for the rating for the overall achievement of the Project Development Objective. Due to the termination of PNPM block grants during project implementation, there is no data for the first Key Performance Indicator on attendance in PNPM meetings, 37

51 yet the PDO of the project is clearly associated with participation in PNPM. On the other hand, the project has performed very well if measured against additional indicators relating to motivation to participate in village planning meetings. Stakeholders (DFAT and Bappenas) suggest that the rating for the PDO takes into account feasibility and cost for replication of the project. In Jackie s opinion, such intervention requires a big investment in the beginning but small amount to continue the support. c) Weaknesses The PNPM Generasi team from KMN and MOV pointed out a few weaknesses of CC II. CC II did not integrate sufficiently with PNPM Generasi. Its creative outputs did not convey substantive information on PNPM Generasi. (Harno, KMN and Bastaman, MOV). The project has not shown direct impacts on health and education. (Bastaman, MOV and Patrick, DFAT) But Bappenas thinks it is too early to measure such impacts given the short implementation period. (Woro, Bappenas). d) Future Replication It is important to integrate and institutionalize such creative approaches into the system to ensure sustainability. (Bappenas). To what extent will the government respond to the voices conveyed through the creative mediums? What would be the ultimate goal? (Woro, Bappenas). Ministry of Villages has already allocated funds for the replication of Creative Communities through the Community Centers it plans to establish at the local levels to provide capacity building for the community. Besides cultural activities, the Community Centers will support the use of Information Technology in village development. The government s challenge is finding a mechanism to channel funds to the local community for the Community Centers. MOV does not need funds but rather technical expertise for the design of the replication of CC II. (Bito, MOV). MOV would like to train facilitators for in the use of creative approaches in facilitating community development. (Bito, MOV). Difference in opinion between MOV and Bappenas regarding replication of Creative Communities. MOV says there is no need to pilot the use of creative approaches again but scale up the intervention through the Community Centers and Village Law facilitators training. (Bito, MOV) Bappenas, however, suggests further testing to obtaining a better understanding of the potential results of using creative approaches. (Woro, Bappenas). Is it better to replicate the creative approaches through formal Village Law facilitators or outside the formal structure through CSOs given the technical expertise required? (Jackie, WB). There is a need for both formal and informal spaces for community to participate in community planning. CC II provides a different atmosphere that is more accessible for the poor and marginalized to participate. It also provides a more conducive environment for the discussion of specific topics like Desa Adat (Customary Village). (Prahastanto, WB). 38

52 CC II activities can be replicated using village funds (APBDes) instead of block grants. Different sectors can be engaged through these activities to address issues that are cultural in nature, such as social taboos relating to health. (Sentot, KOMPAK). Attendance List No Name Organization/Unit 1 Laila Yudiati Australia DFAT 2 Patrick Hermanus Australia DFAT/Generasi 3 Bito Wikantosa KDPDTT (Ministry of Villages/MOV) 4 Sentot S. Satria KOMPAK 5 Gerda Gulo PSF/WB 6 Ikhwan Maulana PSF/WB 7 Natasha Hayward PSF/WB 8 Jackie Pomeroy PSF/WB 9 Amy Sim Kok Eng PSF/WB 10 Wirahman D. Kemendagri (Ministry of Home Affairs) 11 Bastaman KDPDTT (Ministry of Villages/MOV) 12 Suharno KMN (National Management Consultant) PNPM Generasi 13 Ferry Yudisatria KDPDTT (Ministry of Villages/MOV) 14 Woro Srihastuti S Dit PK Bappenas (Directorate of Poverty Reduction) 15 Novi Susanto Dit PK Bappenas 16 F. Prahastanto PSF/WB 17 Festina Lavida PSF/WB 18 Sonya woo PSF/WB 19 Inge S. Tan PSF/WB 39

53 Annex 7. Borrower's Completion Report The Executive Summary of the Project Completion Report (Solvang, May 2015) is included here. Executive Summary The Creative Community 2 (CC2) project was implemented from April 2013 March 2015 by the Kelola Foundation with support from the PNPM Support Facility (PSF). Project Development Objective (PDO) was to improve the ability of villagers, especially the poor and marginalized people, to participate actively in PNPM processes using cultural expressions in project locations. The project was designed to support The National Program for Community Empowerment (PNPM) in addressing challenges in achieving broad based participation for community planning. The project was implemented in two stages: Phase 1 (April 2013 September ) and Phase 1 Extension (October 2014 March 2015). The project was implemented through two main modalities: firstly the Creative Empowerment Process (CEP) where Creative Community Groups (CCG) were formed and trained in using Theater for Development (TfD) or Participatory Video methodologies to explore their situation and express their ideas and concerns; secondly sub-grants were provided to local individuals and organizations to support the use of cultural expressions to stimulate community participation in village planning. Overall, the project focused on creating informal forums and developing engaging facilitation techniques to enable the participation of community members typically not involved in formal development processes. Phase 1 developed and tested the design for introducing creative approaches to support participation in PNPM. The Creative Empowerment Process was implemented in a total of three villages in West Nusa Tenggara, South Sulawesi and West Kalimantan Provinces. A total of 206 (123 women, 83 men) community members directly benefitted from the theater and video activities, which culminated in 17 public events and cultural dialogues held jointly with PNPM Rural and PNPM Generasi. An estimated 2,700 community members were indirectly involved as members of the audience. In addition, eight sub-grants projects were implemented in 10 other villages in the same districts. Lessons from Phase 1 showed that when creative approaches were used in integration with PNPM and formal development processes, there was a significant increase in attendance of the marginal, poor, women and youth. Village level PNPM workers and community members were enthusiastic about the approaches, and the opportunities created for community members to express their concerns and ideas and interact directly with PNPM workers and village governments. By June 2014, a new village law was promulgated, which will dramatically increase the role, responsibilities and resources of the village community. Consequently, PNPM began a transition towards the new law and to meet the demands of the changing policy environment. At the same time, PSF began its own transition towards a hand over of several programs, including CC2 to a Managing Contractor (MC) under the Australian DFAT. This led to a restructuring of CC2 at the end of Phase 1, which reduced the number of locations from 50 to 24, and brought forward the 43 Between July September 2014, the project went through a restructuring process in view of the transition of PSF. During this period of time, the focus of the project was on modifying the implementation plan to reduce the scope and period for the replication of activities. 40

54 closing date of the project to 31 March 2015 instead of 30 June 2015 as originally stated in the approved World Bank Creative Communities II Project Paper. The CC2 Phase 1 Extension was implemented effectively between October 2014 and March CEP activities were implemented in 24 villages in East Nusa Tenggara (NTT), Bali and West Java Provinces community members (546 female, 476 male) were members of video or theater CCGs. The CCGs were supported by 24 facilitators (with experience in video and theater making) and 24 village level creative empowerment cadres (KPK) who were trained in community facilitation using creative approaches. 72 theater and video events were held attracting an estimated 9,600 members of the audience, including PNPM workers and village leaders. In addition, 6 subgrants were provided to local organizations and individuals with the aim of supporting the socialization of the new Village Law or supporting PNPM Generasi to raise awareness on health and nutrition related issues. Lessons from CC2 Phase 1 were documented in an external evaluation (June 2014) and in the CC2 Phase 1 Progress Report (September 2014). An evaluation of the CC2 Phase 1 Extension was carried out in February March 2015 using mixed methodologies including a perception survey in 24 CEP villages, documentation of case studies in CEP and sub-grants villages and facilitators reflection workshops. Key Lessons Learned from CC2 1. Creative approaches are highly effective in engaging the marginal, poor, women and youth. More than half of direct participants in CEP activities were female. Most of the facilitators and KPKs were male. A male dominance could be an obstacle to women s participation, however the high number of female participants regardless, indicates that the approach is well suited to enable women to participate. During the Extension, 43% of participants were classified as poor, 44 72% were marginalized 45 and 36% youth under 17. The latter group was not intended as a target group in CC2 Phase 1. The experience of Phase 1 noted, however, that youth were often not included in village planning processes but they gravitated towards the opportunity to learn new technology and have new experiences. Youth were therefore included as one of the target groups in the Extension % of beneficiaries in CEP Extension reported to be more motivated to participate in formal meetings in the villages after the project. The beneficiaries perceived themselves to have become more active in the community after participating in the project. Figure 1 shows what the beneficiaries perceive they would do before and after the project when they heard about complaints and problems in the community. 53% reported they did nothing before the project, while 6% reported they do nothing after the project. 44 Poor defined as belonging to the 40% lowest household income 45 Defined as non-participants in formal PNPM or village planning meetings in the 6 months prior to CC2 activities 41

55 Figure 7.1: Beneficiaries of CEP in CC2 Phase 1 Extension perceptions about what they would do when there was a problem or complaint in the village before and after CC Percentage Do nothing Discuss with friend/neighb ors Bring to community meeting Other Don't know Before CC After CC Creative approaches were also effective in increasing the agency of CCG members, defined by the project as an increase in self-esteem, ability to express themselves, strengthened networks and a stage for expression and understanding of participation as a right. Figure 2 shows beneficiaries perceptions about the individual benefits of participating in the project. Figure 7.2: Beneficiaries of CC2 CEP in Phase 1 Extension perceptions of what they gained from participating in Creative Communities (in response to open ended questions allowing multiple answers, in percentage) Get to know other Added communities in organizational other villages, 7.82 experience, More activities Can access in the village, important people 18.5 such as KPK, KPMD, village gvmnt, 8.12 More self confident, Gained place and friends to disuss with and express opinions, More aware of my rights to participate in village development, Learn new ways of expressing complaints, Learned new skills/knowledge, 38.2 Understand situation in village better,

56 There were no significant differences in how the poor and non-poor perceived these benefits, indicating that the approaches are also effective in empowering the poor. 3. The use of cultural expressions and creative approaches in formal community planning processes generates community interest and breaks down formal power structures that stand as barriers to the participation of the marginalized. CC2 Phase 1 showed examples of how attendance at PNPM events increased as a direct result of the project. By offering a service to village government in attracting community members to formal meetings, creative communities could generate support for creative communities and increase leverage and opportunities for the marginal to voice their concerns. 4. Barriers to the projects ability to directly support PNPM by directly impacting attendance and influencing facilitation skills of PNPM workers included PNPM s capacity to engage directly with CC2 given the PNPM facilitators workloads, and the lack of formal instructions from the central government (Ministry of Home Affairs) to village level PNPM workers to engage with CC2. CC2 also did not succeed in mainstreaming creative approaches into PNPM SOPs aside from allowing the use of culture mediums in socializing PNPM in the revised PNPM Rural Technical and Operational Guidelines (PTO). Similar challenges may be anticipated when using creative communities in the future to support the implementation of the new village law. 5. During CC2 Phase 1, the use of cultural activities in integration with formal PNPM meetings had a significant positive impact on increasing the attendance of community members who typically would not attend. Due to the PNPM transition such integration could not be replicated in the Extension. There is nevertheless evidence that the CEP process and sub-grants overall created new opportunities for community members to get together and discuss their ideas and concerns, and meet with village leaders and stakeholders to express their opinions. There is also anecdotal evidence that the CCG groups were able to directly influence decision making in some villages through their use of theater and video. 6. For creative communities to have an impact on active participation on villagers in development processes joint planning between the project and village level stakeholders is necessary to create the integration required between the project and formal development processes. The lesson from CC2 is that sufficient time needs to be allocated to such joint planning. Where integration with formal village planning processes occurs, creative approaches are highly effective in increasing attendance at formal meetings, motivating people to participate, and in bringing the voices of the marginal and poor into decision-making processes. 7. Creative approaches can be used to impact different levels of participation from attendance to knowledge about how to participate, motivation to participate, access to arenas of decisionmaking and control in the decision-making process. For a project that aims to increase communities ability to participate, it is important to define and operationalize the objectives of the project clearly, and select appropriate indicators to reflect the PDO. The CC2 PDO includes active participation. However, increased attendance at PNPM meetings as a KPI proved ineffective in measuring the real impact of the project on community empowerment given the gap between attendance and actual participation. The M&E plan for the Extension added indicators for empowerment to bridge this gap. In terms of influencing attendance, the evaluation shows that attendance at village level meetings was often closed and structured 43

57 rather than open, and hence could not be easily influenced. The baseline highlighted a larger opportunity to impact attendance at sub-village level (i.e. dusun or RT/RW levels) or on improving quality of representation rather than mass attendance. Timely assessments and baselines can provide the information about the barriers and opportunities for active participation of the community needed to design effective project strategies. 8. Development initiatives often aim to change dynamics of decision-making in a community by enabling more people to participate. Particular groups in the community may see this as a threat to their own access to resources. As such, development initiatives are political and often contentious. 46 When implementing the CEP, CCGs may become a stakeholder positioned along the fault-lines of pre-existing tensions and conflicts in the communities. This was documented in case studies from e.g. Barang and Ngabheo Villages during the Extension. The CEP will likely be most effective in achieving influence over decision-making in villages that have leaders who are committed to broad and active participation. Management of the risk of potential backlashes against project participants with the involvement of village leaders is needed to ensure that no harm is done. 9. When Creative Community Group (CCG) members have ownership of the mission and vision of the group, it is more likely that the group will without support after the project ends. Assessments at the outsight helps to identify key groups and individuals in the community that are likely to have a vision and mission for community engagement and/or creative skills. The evaluation of CC2 shows that in communities like Panji (Bali) and Malaka (Lombok) where the communities have a strong motivation driven by key members, they also have the ability to find resources needed to continue their activities independently. CC2 also adopted strategies to support CCGs in making future strategies and plans and the sharing between creative communities of different villages. 10. The Evaluation of the CC2 Phase 1 Extension CEP showed that the lack of strategic engagement with formal development processes that could have increased the CCGs success in influencing policy and decision-making. Participatory social analysis generated issues and themes for video and theater, however too much focus on the media also influences the agenda, e.g. where issues not suited for video were left out or focus was issues causing family tragedy as that makes good theater. A more open ended approach to advocacy with less focus on predetermined media and project driven video screenings and theater performances, would have allowed for a mapping of priorities and development of strategies for how to deal with the most important issues effectively. Linking these strategies directly to formal processes of village planning and decision-making would likely have increased their effectiveness in achieving higher levels of active participation. 11. The strengthening of the main Grant Recipient s (i.e. Kelola) capacity for M&E led to a systematic analysis of the barriers to community participation linked to the articulation of the project s theory of change and the design of logical frameworks and M&E plans that improved the management of project results during the Extension, and enabled the measuring and documentation of project impact. Such evidence based documentation allowed the project 46 Invalid source specified. 44

58 management to build a case for the effectiveness of creative approaches for future development initiatives, such as the implementation of the new village law. 12. The CEP model implemented in CC2 is intensive and requires resources. Better integration with formal development processes will make the intervention more efficient, and will provide CCG members with insight into the workings of formal planning processes, and an experience in accessing and influencing decision-making that is likely to motivate more active participation in the future. 13. Sub-granting is an effective modality to mobilize communities and socialize messages. Subgrantees use of cultural events and activities created broad participation and opportunities to socialize messages about PNPM processes, health and the new village law. E.g. during the Phase 1 Extension, Lite Institute in Bali, engaged hundreds of local cultural workers (e.g. dancers, musicians and craftspeople) in a village festival that pulled a large audience to a socialization of the new village law. The most effective sub-grantees also succeeded in creating opportunities for communities to interact directly with PNPM workers or representatives of village government to address issues of concern. E.g. during Phase 1 Extension, Jatiwangi Art Factory (West Java) facilitated community members in making community television that allowed them to meet directly with PNPM Generasi and village leaders. The television production also brought the dialogue into people s homes, which created new forms of dialogue about community health issues The Sub-grant model is best implemented based on an understanding of partners capacity to deliver quality programs to the communities. Lessons from CC2 show that grassroots organizations, especially in remote areas, could benefit from capacity building, not only on administration, finance and report writing, but also community facilitation and technical skills. Implementation timeframes should allow for such capacity building activities with subgrantees. Future capacity building of sub-grantees could be integrated with the CEP training to make effective use of training modules and materials developed for effective community facilitation. Key recommendations for Future Creative Communities Projects Results-based Management 1. Ensure that results-based management and M&E and learning frameworks are built into the project, for example, by using appropriate logical frameworks and indicators and incorporate a MEL plan into day-to-day management and implementation of the project. When working through an implementing partner, ensure the partner has effective project cycle and result-based management capacity. Design 2. In the design phase, conduct an analysis of entry points and barriers to the project s ability to influence formal development processes, both in terms of attempting to mainstream creative approaches within government at the national level, or for project participants to gain access to and influence over local development planning processes. 47 Creative Communities 2 Evaluation Report on CC2 Phase 1 Extension Sub-grants (Annex 3) 45

59 3. Articulate strategies for using Creative Communities activities for the communication of key messages as a service to authorities, and for community mobilization to build positive relations and leverage between its participants and village leaders. 4. Take a wider approach to the use of media (e.g. theater and video) in empowerment to stimulate active participation that leads to influence over decision-making processes. Key issues generated through a social analysis require strategy for how to best create a wanted change that benefits the marginalized. Too much focus on the media may lead to the question: What should we make theater/videos about? Rather than: what are our key challenges, and how to change them? The latter being the most relevant to active participation. Future projects should let the issues guide the making of effective strategies for wanted change. 5. Continue sub-grants as a modality to support local NGOs, CBOs and individuals. using creative approaches to stimulate broad participation. Allocate sufficient time and resources that allow for capacity assessments and the capacity building of sub-grantees to ensure that they are able to delivery high quality activities and results. The provider of sub-grants (in this case Kelola) needs to have Partnership and Capacity Building Specialists in their sub-grants management structure. Capacity building of sub-grants partners may integrate with CEP capacity building for community facilitators for increased efficiency. Implementation 6. In the inception phase, conduct a timely baseline study to identify who are marginal and poor and barriers to and opportunities for participation to inform implementation strategies and facilitators manuals, and conduct village level assessments to identify people and groups that share a vision of community development, and community leaders committed to broad participation. 7. Start implementation with joint strategy and planning, including joint risk analysis, with authorities at all levels, including village level, to identify potential for integration of the project with specific formal development processes. 46

60 Annex 8. List of Supporting Documents Government of Indonesia. (2014). Undang Undang Nomor 6 Tahun 2014 Tentang Desa. Gultom, Y. M. (December 2013). Laporan Hasil Evaluasi Percontohan Program Komunitas Kreatif II: Proses Pemberdayaan Kreatif di 3 Kabupaten di Indonesia. Migunani. (2015). Creative Communities II Final Evaluation: Summary of Perception Survey. Yogyakarta. PNPM Support Facility. (April 2012). Project Proposal: Creative Communities II -- Preparation and Pilot. Pollock, I. F. (March 2015). Summary of the Baseline Survey: Creative Communities II. Simatupang, G. L. (March 2013). Creative Communities II: Field Assessment. Simatupang, L. G. (2012). Evaluasi Proyek Pilot Inisiatif Komunitas Kreatif. Sloman, A. (September 2014). Creative Communities II: Phase 1 Progress Report. Solvang, I. (May 2015). Creative Communities II: Project Completion Report. Jakarta, Indonesia. Taylor, G. a. (March 2014). Review of evaluation approaches and methods used on interventions in women and girls' economic empowerment. ODI. Widuri, D. E. (August 2014). Delapan Studi Kasus Komunitas Kreatif II. World Bank. (2004). ICM JSDF-INDONESIA: WIDOWS AND POVERTY. World Bank. (August 22, 2014). Paper on a Propoposed Project Restructuring of the Creative Communities II. World Bank. (December 13, 2012). Indonesia - Country Partnership Strategy , Report No World Bank. (July 22, 2008). Country Partnership Strategy , Investing in Indonesia's Institutions, Report No IND. World Bank. (June 13, 2014). Paper on a Proposed Restructuring of the Creative Communities II. World Bank. (June 2012). Grant Reporting and Monitoring (GRM) Report, Creative Communities Fund, JSDF TF World Bank. (May 2013). Project Paper for Small RETF Grant to Yayasan Kelola for the Creative Communities II. World Bank. (November 16, 2007). JSDF Grant for Creative Communities Fund Project Grant Number: TF World Bank. (November 8, 2012). Project Appraisal Document, National Program for Community Empowerment in Rural Areas

61

62 MAP 49