PNPM SUPPORT FACILITY (PSF) Project Proposal. PNPM Monitoring and Evaluations (M&E) and Special Studies ( )

Transcription

1 PNPM SUPPORT FACILITY (PSF) Project Proposal Project Title: Development Objectives: Key Performance Indicators PNPM Monitoring and Evaluations (M&E) and Special Studies ( ) The objectives of the PNPM M&E trust fund program are to: (i) provide stakeholders with empirical data regarding results and impact of PNPM Mandiri; (ii) conduct research in greater depth regarding special topics of concern to PNPM and the field of social development; and, (iii) support capacity building efforts for Indonesian research organizations. Data and findings used to improve implementation and enhance design/impact in existing and new programs Evidence and recommendations regarding sustaining povertyreduction and pro-poor growth impacts of CDD programs delivered to PNPM stakeholders Evidence and recommendations regarding PNPM, local economy and livelihoods delivered to stakeholders Mapping of community groups contributions to accountability in achievement of minimum standards service delivery, delivered to stakeholders Evidence regarding local constraints to adopting PNPM governance practices delivered to practitioners forums Recommendations regarding strategies for increasing inclusion and outreach to marginalized households and groups delivered Research activities, products, and services planned for and developed in collaboration with PNPM/Cluster 2 policy-setting stakeholders Number of studies completed as part of the round of evaluations Analytic pipeline proposal for future work to address new frontiers/special topics of concern to PSF stakeholders Capacity-building action plan for Indonesian research partners developed and implemented Short-course training modules focused on technical details in implementing qualitative and/or quantitative social science research delivered Executing Agency: PSF Secretariat/World Bank Estimated Budget: US$ 4,750,000 Estimated Duration: 32 months (covering activities to December 31, 2015) Page 1 of 25

2 Geographic Coverage: Implementation Arrangements: PSF Secretariat Points of Contact: PSF JMC Approval Sought: National PSF will use specified funding from the PSF Trust Fund to issue contracts to qualified research organizations and/or consultants to carry out the analytic works. Natasha Hayward, Senior Social Development Specialist Approval from the JMC is sought to allocate $4,750,000 in funding from the PSF Trust Fund to proceed with the activities associated with this Project Proposal. This proposal, for a subset of key evaluations and pieces of analytic work for the PSF portfolio from , will support the continued production of high-quality evidence and well-researched recommendations to inform both operational procedures in, and policy dialogue regarding, Cluster 2 of the Government of Indonesia s (GoI) poverty program, in particular, the PNPM Mandiri program. The analytical agenda focuses on GoI evaluation concerns relating to community empowerment and poverty reduction as well as the strategic directions outlined in the PNPM Roadmap, the MP3KI and elsewhere. The analytical agenda led by the PSF s Monitoring and Evaluation (M&E) team, developed in collaboration with GoI, has consistently emphasized the rigorous construction of the PNPM and community-driven development (CDD) evidence base through empirically-focused impact evaluations and quantitative studies as well as detailed case studies and longitudinal qualitative work (please see Annex 1 for fuller details of recent studies/findings completed ). The M&E team s core program continues that approach with a focus on quality, relevance and timeliness of evaluations and analysis, while expanding the range of its partnerships with both national and international institutions in the delivery of M&E products. I. Background 1. The GoI s National Program for Community Empowerment (PNPM) program is built around the core hypotheses that through combined investment in the institutions, assets, and capabilities of the poor, resulting community empowerment can produce more effective, just and inclusive poverty reduction. 2. Fourteen years after PNPM Rural s inception (as the Kecamatan Development Project) its core objectives remain the same: continuing to invest in community empowerment as a means to more effective poverty reduction. The evaluation work accompanying the PNPM initiative as well as learning-by-doing through the program s operations has delivered much in both practical and theoretical knowledge. Evidence confirms that PNPM Rural raises incomes, reduces poverty rates, and increases access to services, especially in poorer or worseoff households and communities. Evidence also shows that, across Rural and Urban areas, PNPM has effectively addressed part of the national deficit in tertiary infrastructure with lower costs and very high technical standards. 3. Policy challenges remain: marginalized groups (other than the poor) are less likely to participate in the program; good governance practices in PNPM have not always spilled over into Page 2 of 25

3 other local administrative or government spheres; impacts on households in less poor kecamatan are limited; both PNPM Rural and Urban have not been able to address the constraints to longterm, productive livelihoods in all households and communities; and PNPM Rural s massive scale-up (over an abbreviated period) has produced a number of challenges in both scale and scope in the managerial, quality control, and operations fields. PSF analytic work: what difference does it make? Over the years, findings from these studies and the research evidence base have: Informed and influenced policy dialogue and decisions Presidential decision to scale up KDP as PNPM National in 2007; and subsequent commitment by present Administration to sustain PNPM as its core vehicle for community-based poverty reduction, at national scale through 2015 PNPM Roadmap data and evidence contributed to ongoing Working Groups deciding on how to customise/tailor PNPM implementation and block grant formula allocation, according to need (based on evidence of where most impact is seen; and utilizing census findings) RUU Desa evidence (e.g., on critical role of accountability mechanisms at local level) provided to inform/influence content of draft village law currently being prepared for parliamentary approval; Calculation of a supply-readiness index for basic health and education services, which when combined with data on use and access to services is being used by GoI to prioritize areas for PNPM Generasi expansion for Influenced/led to innovative pilots PNPM Peduli s innovative model of partnering with CSOs to better target and empower the most marginalized groups Creative Communities approach of utilizing and reinterpreting traditional media and forms of cultural expression to reinvigorate and promote participation and voice in local level decisionmaking Bappenas intention, with PMD/PU, to pilot a livelihoods-focused PNPM model, addressing jobs and income under MP3KI Generasi s testing of an ECD indicator in three districts and the upcoming introduction of a "nutrition sensitive" indicator beginning in 2014 Contributed to operational adjustments and lessons Mainstreaming of Generasi pilot s performance-based design to all project locations; PNPM Rural s policy of 80% audit coverage Generasi s scale up locations determined according to evidence of where the program is most effective Operational guideline that one woman member is mandatory on the local Procurement committee Facilitator workload analysis to inform assignment of facilitator roles under PNPM Roadmap plans Page 3 of 25

4 4. The GoI has committed to supporting PNPM through 2014 and has developed a Roadmap for the initiative to provide guidance for that period and beyond. A major pillar in the GoI s Roadmap is the mainstreaming and integration of PNPM operating principles into local government, administrative, and service delivery operations. Quality of service delivery and accountability relationships are critical challenges especially in Indonesia s ongoing decentralization process and the shifting equilibrium between central and local authority and responsibility. Currently, Indonesia, while a lower middle-income country, has poor public service delivery standards (and outcomes) far below regional peers, income peers, and even some highly disadvantaged low-income countries. In addition to these challenges, issues are emerging in the practice of governance in the national-scale PNPM program and in demands for attention to economic opportunities and livelihoods development for the poor via the PNPM platform, as described in the Masterplan for the Acceleration and Expansion of Poverty Reduction in Indonesia (MP3KI) document. 5. The PSF M&E framework for PNPM (and associated pilots and initiatives) therefore supports the GoI s Roadmap and other commitments to maximize community-led development at the local level by linking operational experience and local-learning with applied research and evaluation, all in the service of a knowledge base of best-practice solutions for effective CDD and community-led programs in Indonesia. The M&E team s analytical agenda will focus on research and solutions that aim to measure and increase such programs pro-poor impacts and facilitate a better understanding of social and institutional dynamics at the local level and their influence on community empowerment and development strategies and policies. 6. This top-up funding proposal covers a limited set of activities from this larger, overall agenda. This request covers quantitative and qualitative evaluation pieces that have been identified as crucial for effective evaluation of present PNPM/CDD programs. It was agreed with development partners that this evaluation work should be prioritized for action while allowing the special topics agenda to crystallise based on donor and GOI priorities (e.g., as plans for increased focus on rural livelihoods are articulated; as plans for follow up to the PNPM Urban evaluations are agreed, etc). See Section IV below for further detail on the subset of activities covered by this top-up funding proposal. 7. A specific exercise will therefore be undertaken in the second half of 2013 with all stakeholders, to assess needs and to critically review and redefine future frontiers for PSF s analytic pipeline going forward, beyond these core evaluation commitments and in the light of the new PSF strategy, under preparation in May-June This review will result in a revised set of hypotheses and an overarching Analytic Framework for , for which additional resources may be requested in due time to cover any additional resulting plans. 8. The Framework will be reviewed annually as a reference from which to prioritise and address agreed questions and related studies. It is anticipated that this review and redefinition exercise will take stock of the key trends and changes that have occurred in Indonesia s demographic, political and socio-economic landscape over the last 15 years, as also embodied in the Government s forward looking Roadmap for PNPM and MP3KI Masterplan for Poverty Alleciation. At the same time as consulting and assessing stakeholders latest priorities, the Page 4 of 25

5 review would also critically revisit some of the influential conceptual frameworks used as a reference point in the evolution of the PNPM platform and pilots, identifying any related analytic gaps and future horizons for exploration in the portfolio (e.g., livelihoods development; frontline service delivery, etc). 9. The PSF M&E agenda, developed in collaboration with GoI, has consistently placed equal emphasis on rigorous impact evaluations and quantitative studies as well as detailed case studies, longitudinal qualitative work, and special studies. The core agenda approaches the construction of the evidence base in the same way, and the M&E group and partners believe quality and depth of research rather than methodology are paramount. For that reason, the push to include more national partners will be accompanied by an emphasis on quality control tailored to a research partner s needs and which evolves as skills, capacities, and personnel evolve. 10. More joint outputs with national research bodies and GoI task forces will be pursued; enhancing partnerships with both international and Indonesian universities and think tanks; and active collaboration with Indonesian research firms and groups, all of whom are expected to absorb more analytical and research services and products in the years to come. II. Project Development Objectives 11. The M&E agenda was originally developed to: (1) provide stakeholders with empirical data on program impacts and outcomes in PNPM communities; (2) plan and conduct research in special topics of concern to those in PNPM operations as well as stakeholders more generally; (3) support capacity building within, and skill transfer to, Indonesian research organizations. These objectives remain relevant and it is expected that they will foster an environment which encourages research in social development and/or community-led development in Indonesia. III. Project Description 12. This Project consists of the following components: i. Monitoring and evaluation (M&E) activities to provide stakeholders with empirical data regarding the results and impact of PNPM Mandiri; ii. iii. Research in greater depth undertaken regarding special topics of concern to PNPM and the social development field in Indonesia; Capacity building to enhance ability of Indonesian social science research organizations to conduct M&E and special studies. 13. PSF s current strategy 1 formulates four action-oriented pillars to guide operational and analytic activities: continued poverty impact, improved accountability in service delivery, 1 Note that the 2012 PSF Strategy is undergoing revision and an updated Strategy will be presented for JMC approval in July The present strategy, and its associated Analytic Framework, has served as the reference point for the core analytic questions covered in this M&E proposal. As noted in paragraph 8, a review and Page 5 of 25

6 social justice and inclusion, and a supportive enabling environment. These pillars also serve as the organizational frame behind the M&E analytic agenda contained in this proposal (see Table 1 below and Annex 2 for further detail on this organizing framework, associated research questions and resulting evaluation topics). 14. The main organizing questions are: a. Continued poverty impact: How, why and where does PNPM reduce poverty and how can poverty-reduction impacts and rural livelihoods development be enhanced? b. Improved accountability in service delivery: In what ways and under what conditions can community participation improve local governance and service delivery for all? c. Social Justice and Inclusion: What does PNPM need in order to be more inclusive and enhance opportunities and empowerment of women, the very poor, and the marginalized? d. Supportive Enabling Environment: How can the proposed reforms relating to PNPM and other poverty-reduction or pro-poor growth initiatives be supported by and coordinated through an appropriate combination of policies, institutions and operational systems and capacities? Table 1 Organizing Framework and Guiding Research/Evaluation Questions i. Continued poverty impact: How, why and where does PNPM Rural reduce poverty? 1. Related Research Questions Policy/program relevance a. How are local economies, productivity and rural livelihoods affected by PNPM infrastructure/investments? How to adapt PNPM for further livelihoods development? In what contexts does PNPM work better? How should the program be geographically b. Is PNPM as an instrument fulfilling its stated social and economic objectives? What is the role of Cluster 2, visa-vis Clusters 1 and 3? Can PNPM support GoI s social protection efforts? c. What is PNPM s effectiveness in Indonesia s most disadvantaged/poorest regions? (see also Q2.) (e.g., evaluation shows greatest poverty impact in poorer areas/households, but implementation challenges and bottlenecks are greatest in poorest and most remote areas) d. Which aspects or features of PNPM are performing better? How does the PNPM delivery mechanism for particular outputs/investments compare to other (government) options? What poverty reduction can be attributed to PNPM? targeted? Crystallizing PNPM s role and efficacy in the logic of GoI s poverty strategy, mediating between C1 and C3, as a platform for accountability in delivery of programs/services. Optimizing PNPM effectiveness where needs and challenges are greatest. Should PNPM design, activities and indicators be adjusted for regional/socio-cultural variation? Identifying the most effective parts of the PNPM process/delivery mechanism. Continued justification of program as core of Cluster Improved accountability in service delivery: How does community participation and empowerment transform local governance and service delivery? redefinition exercise will take place later in 2013, to update an Analytic Framework for , ensuring consistence with the updated PSF Strategy 2013, and including any special topics/research questions or thematic studies not covered in this core proposal. Page 6 of 25

7 2. Related Research Questions Policy/program relevance a. What are local understandings, pathways and practices of accountability in Indonesia s diverse contexts? b. Why do/don t PNPM-specific governance processes translate into other broader local governance/service delivery arenas? Why is it hard to institutionalize accountability and participation? What impacts do broader processes of decentralization and politics have on space for empowerment through PNPM? c. What can the PNPM platform do and how far can it be exploited? Where are communities most effective and what are their limitations? Identifying effective mechanisms to foster better downward accountability and transparency, including channels for recourse and grievance redress. Identification of conditions for positive synergies between PNPM and village government. Identification of PNPM s limits, and wider pressure points beyond the program. Identifying and rebalancing asymmetries of information in the rural space. Identifying key community-driven practices/institutions/functions to be institutionalized within local planning mechanisms. What roles can communities play in co-production/co-delivery of services? d. What encourages sectoral and LG Identifying proper incentives. Informing debate on partnership? what responsive governments can deliver. 3. Social justice and inclusion: How can the PNPM platform be more inclusive and foster empowerment of women, the poor and marginalised? 3. Related Research Questions Policy/program relevance a. Do elites capture or contribute to PNPM? Where, how and why? Identifying design elements that will lower capture or effectively utilize elite capacities b. How well does PNPM target the poor and marginalised? Why can t some groups participate? What can we learn from other Identifying design/implementation reforms that will promote more inclusion/greater participation and support social justice for typically excluded groups. more targeted delivery mechanisms and modalities (Pekka, Peduli, DPOW, etc)? c. Does PNPM address the needs (institutional, social, financial) of poor rural women? What can we learn from other delivery mechanisms and modalities (Pekka, Peduli, Justic, etc)? d. What is PNPM s effectiveness in Indonesia s most disadvantaged/poorest regions? (see also Q1) How can targeting of women be improved? What should be goal and design of RLF? Should PNPM design, activities and indicators be adjusted for regional/socio-cultural variation? 4. A supportive enabling environment: How can proposed reforms be supported by and coordinated through an appropriate combination of policies, institutions and operational systems and capacities? 4. Related Research Questions Policy/program relevance a. What are the PNPM core functions, systems, principles to be sustained under any vision of more integrated local and community driven development? b. What are the institutional and policy arrangements (at central and local levels) that would support proposed reforms? What are the Determination and agreement on those benefits (institutions, practices, etc) to be sustained under the future vision How do existing formal/informal local institutions contribute to en enabling environment for inclusive development and local governance? Identifying a Page 7 of 25

8 institutional impediments to sustainable local development? (see also 2(b)) How do nongovernmental actors (NGOs, CSOs, staff) mediate/support PNPM s effects? c. How to institutionalize technical capacity in M&E (especially qualitative and formal IE methodologies) outside of PSF? joint framework for village level institutions (inc. options for facilitation) to support inclusive local development. Independent research institutes/university department/think-tanks with sufficient capacity for applied and longer-term social analysis to inform GoI s poverty programming. 15. Guided by the framework and research questions above and in order to provide policyrelevant evidence, this Project directly supports evaluation plans linked to key Cluster 2 projects: (a) PNPM Rural and the related PNPM Mandiri Roadmap; and, (b) PNPM Generasi. The main analytical projects are: a. PNPM Rural and Roadmap Evaluations: Economic Impact Simulation This exercise will demonstrate the shape and size of the benefits PNPM Rural brings to the Indonesian economy and rural livelihoods. It will compare the PNPM way of addressing local infrastructure and productive capital needs to a simulation of the next-best alternative for addressing those same needs. By cataloguing the local productivity gains, contributions to local economies and livelihoods, and cost savings for the central government, the Economic Impact Simulation will be a comprehensive summary of the measurable financial advantages of PNPM principles and an identification of the most effective parts of the PNPM process/delivery mechanism in supporting local economic gains. Sentinel Villages The aim of the Sentinel Villages project is to better understand the changing nature of community experience and satisfaction with, as well as desires for, the PNPM program and community members participation in the program. Understanding the way these experiences evolve will in turn lead (in real time) to suggestions for changes in program implementation. The methodology will combine repeated qualitative work and analysis over a number of years with a quantitative analysis based on those same longitudinal qualitative records. Research foci will include the changing nature of participation, social capital, and the evolution of democracy at the local level and its impacts on PNPM s value as a community solution. Results will directly support PNPM policy and Roadmap elaboration, especially those policies focused on engaging local government effectively in poverty reduction programs and other pro-poor development initiatives. Accountability initiative: This project will take further the question on what enables or constrains PNPM-type good governance and accountability practices to be taken up more widely within the community or local government or administration, and how this relates to service delivery/local government performance at the local level. Based on global and local experience, a series of propositions for testing key hypotheses linked to effective triggers for improved accountability at the local level are being developed. This piece will be linked to the PNPM Roadmap and wider local governance initiatives under PSF which may include: the planned Enhanced Page 8 of 25

9 Empowerment Experiment (EEE) pilot which will test mechanisms for improving participation and accountability; the Community Legal Empowerment project (in partnership with Pekka) and the overall frontline service delivery package supported by Ausaid. Revolving Loan Fund (RLF) Expenditure Tracking exercise The Revolving Loan Fund initiative has delivered business-oriented credits to every community with PNPM block grants. However, very little is yet known about what local enterprises or households these credits supported. The GoI would like the RLF initiative to be an effective vehicle for giving both savings and credit access to marginalized households and communities as part of its rural livelihoods strategy, but has a limited evidence or knowledge base to consult for RLF-centered policy deliberations. This tracking exercise is seen as the first building block in the construction of an RLF-centered evidence base and as a complement to the ongoing RLF pilot exercise (and associated evaluation). Primary quantitative research questions will include: What are the characteristics of an RLF beneficiary? What are the characteristics of an RLF-group manager? To what uses were RLF credits put? How did beneficiaries treat RLF credits (i.e, as actual credits or something closer to a consumption-smoothing vehicle)? The evidence gathered can provide preliminary policy direction as well as provide impetus for the next round of analytical and research activities focused on the RLF initiative. Gender, livelihoods and empowerment: Having supported a baseline evaluation for the PEKKA program a community-driven initiative targeting poor widows and female-headed households PSF will continue its support to PEKKA s evaluation by assisting PEKKA with analysis of available datasets to determine the program s quantitative impact, and preparing a program of associated qualitative work to understand what is behind PEKKA successes and challenges, learning lessons of importance on gender, livelihoods and empowerment of key relevance to the PNPM approach to community empowerment and poverty reduction. b. PNPM Generasi Evaluation: Quantitative Impact Evaluation: Building on the previous Generasi impact evaluation (over the years ), and with the benefit of controls largely having been maintained in the original pilot kabupaten, a feasibility study is planned to ascertain whether a repeat fourth round of the IE survey instruments may be possible. This is planned to examine impacts and results from an enhanced Generasi design in new provinces and to continue to generate longitudinal panel data on Generasi's impacts in areas which continue under the program. With MCC also fielding a formal Impact Evaluation in their 3 provinces, the team will also provide technical and coordination inputs to that exercise. Qualitative studies: the Generasi team is planning qualitative pieces in order to complement any quantitative evaluation and better understand how accountability between community members and service providers is created and sustained. These studies, which the PSF M&E agenda will also support, may include: (i) lesson learning case studies detailing the interactions between community members and Page 9 of 25

10 frontline service providers; (ii) in those areas where Generasi will phase out in 2013, a prospective study summarizing the sustainability of community capacity that was built up through six years of participation in PNPM Generasi. 16. Continuing best practices from the past, this Project will use a number of different approaches to answer the questions framed by these core evaluation pieces. Methods that may be used by the Project include: Impact evaluations to assess the effects of operations; Experimental interventions to test and empirically assess how altering elements of programs can enhance outcomes; Qualitative and ethnographic studies to explore in greater depth the factors explaining program performance or other presenting issues of concern; Quantitative and spatial analysis to understand and assess sources of variation; Stocktaking and longitudinal reviews on specific themes and program areas. 17. The Project will also develop mechanisms and initiatives to strengthen capacity in local social research organizations to deliver high quality analytical work (see implementation arrangements below). The core M&E team is relatively lean it is a group of 4 fulltime specialists so the team will link study and research implementation with a capacity building strategy to enhance the ability of Indonesian social science research organizations (including both universities and independent think tanks) to conduct M&E and other special studies. To date, this has largely been addressed through an implicit partnership mode, with all studies executed alongside a local research institution, building methodological training goals and field-level quality assurance into each activity. This modality has demonstrated its limits, so the team is planning a more explicit set of capacity-building and skill-transfer activities, primarily to address local research institutes capacities for qualitative methodologies and quantitative impact evaluation skills. 18. Coordination of plans and dissemination of analytic activities across interested Cluster 2 partners and beyond will be a priority. A primary point of reference for sharing of plans, with opportunity for cross-coordination and inputs, and as a venue for dissemination of findings will be the PSF JMC. In addition, a further mechanism (e.g., an Analytic working group) for practical coordination purposes will be agreed with the Technical Committee of the JMC, in order to function as a forum with regular meetings and virtual opportunities for sharing and discussion. 19. For dissemination, in addition to the more conventional channels, such as presentations to key stakeholders/institutions and publication of related reports and digestible summaries, all materials will be made available via the PSF website. The team recognizes however, that such channels remain limited and so, for major studies, or sets of studies, tailored dissemination plans will be prepared and the JMC and analytic coordination group will serve as advisory boards to provide inputs on new audiences and channels for dissemination. It is also proposed that early drafts of Concept Notes, draft reports, presentations, etc are shared for comments with members of the working group, to give an opportunity for inputs and reactions throughout the process and not simply at the conclusion of each study process. Page 10 of 25

11 IV. Project Outcomes and Deliverables 20. The results indicators in the table below were developed specifically to correspond to activities covered by this top-up funding request. For more detail on the links between the body of evidence so far accumulated from completed PSF M&E research and the ongoing program, also see Annex 1. PDO Level Result Indicators Unit of Meas ure Base line 2012 Cumulative Target Annual Targets (cumulative) Data Source/ Methodology Project Development Objective(s): (i) to provide stakeholders with empirical data regarding the results and impact of PNPM Mandiri; (ii) conduct research in greater depth regarding special topics of concern to PNPM and the social development field in Indonesia; and, (iii) support capacity building efforts for Indonesian social science research organizations (i) To provide stakeholders with empirical data regarding the results and impact of PNPM Mandiri Data and findings used to improve implementation and enhance impact in existing and new programs. Evidence and recommendations regarding sustaining poverty-reduction and pro-poor growth impacts of CDD programs delivered to PNPM stakeholders (Continued Poverty Impact) Evidence and recommendations regarding PNPM, local economy and livelihoods delivered to stakeholders (Continued Poverty Impact) Mapping of community groups contributions to Findings incorporated in programs/projects design and implementation documents Policy-focused recommendations identified and delivered to stakeholders Policy-focused recommendations identified and delivered to stakeholders Graphic & summary guide to accountability Findings of completed studies available and adopted by relevant task teams Rural Infrastruct ure Census; PNPM; Respek Evaluation PNPM Respek Evaluation - Findings of completed studies available and adopted by relevant task teams PNPM Rural Impact Simulatio n PNPM Rural Impact Simulatio n; RLF tracking PNPM- Generasi Eval. Findings of completed studies available and adopted by relevant task teams Sentinel Villages report PEKKA evaluation Accountab ility Evaluations/stu dy reports; project documents; task team feedback Evaluations, analysis, reports Evaluations, analysis, reports Evaluations, analysis, Page 11 of 25

12 accountability in achievement of minimum standards service delivery, delivered to stakeholders (Improved Accountability in Service Delivery) Evidence regarding local constraints to adopting PNPM governance practices delivered to practitioners forums (Improved Accountability in Service Delivery) Recommendations regarding strategies for increasing inclusion and outreach to marginalized households and groups delivered (Social Justice and Inclusion) Research activities, products, and services planned for and developed in collaboration with PNPM/Cluster 2 policy-setting stakeholders Number of studies completed as part of the round of evaluations 2 % of produ cts # of Evalu ations /Studi es relationships at the desa-orbelow- level Summary note describing how, where, and why community groups can successfully change operating procedures to their benefit Policy-focused recommendations identified and delivered to stakeholders 100% of studies developed with stakeholder input LLI3 report; - Sentinal Villages report PEKKA evaluation initiative Accountab ility initiative; Sentinal Villages report PEKKA evaluation 100% 100% 100% (ii) To conduct research in greater depth regarding special topics of concern to PNPM and the social development field in Indonesia Analytic pipeline proposal for future work to address new frontiers/special topics of concern to Report summarising critical lessons, analytic gaps and key future Review and redefinitio n exercise completed 2 Including LLI3 and Respek evaluations financed under the existing M&E TF allocation reports Evaluations, analysis, reports Evaluations, analysis, reports Preparatory meetings; advisory board memberships Completed reports published on website Consultations and review report Page 12 of 25

13 PSF stakeholders research questions/horiz ons for PSF (iii) To support capacity building efforts for Indonesian social science research organizations (Supportive Enabling Environment) Capacity-building action plan for Indonesian research partners developed, endorsed by JMC members, and implemented Short-course training modules focused on technical details in implementing qualitative and/or quantitative social science research delivered NA # of cours es held NA Indonesian research partners capacities increased Ensuring a supply of basic social science research skills and upgrading existing embodied capacity Capacity building actionplan endorsed by JMC members Course designed Capacity building actionplan resourced and implement ation started 1 course held Capacity building plan implement ed 2 course held Procurement contracts, progress reports Training modules and curricula V. Implementation Arrangements 21. The Project will be led by the PSF M&E team who will be in charge of directly developing or overseeing development of the conceptual framework for each piece of analytical work, leading the implementation of the works in partnership with local research institutes, and providing quality control of data collection, analysis and reporting. With a renewed focus on partnership, the team will also explore some more innovative delivery models for discrete studies and be pro-actively engaging relevant policy-level stakeholders in leadership of discrete pieces or in contributing to concept development, review and dissemination stages for each piece. 22. While capitalizing on PSF in-house team strengths, the team will work in partnership with local research institutes and other parties to carry out the Project. This will be part of the team s capacity building strategy. The overall approach of this strategy is intended to be consistent with the AusAID-financed Revitalising Knowledge Sector strategy in Indonesia a longer-term effort that will address both individual and organizational capacity in social science research capacity and to build on existing collaboration with the World Bank s PREM Poverty team. The PSF M&E effort, in delivery of this proposal, is intentionally selective in scope, in acknowledgement of the skill-set, capacity and other commitments of the team. Key elements to be addressed in the immediate phase include: i. Short-course training modules in qualitative methods: delivered in partnership with local resource institutions, before handing over full responsibility to those partners. The goal of the design and offering of such short-course trainings would be to ensure basic analytical skills, familiarity with up to date theoretical and methodological approaches, and ability to use these, primarily in applied qualitative research among Page 13 of 25

14 those already engaged in social science research, but with need to enhance staff skills. A potential module/partnership for quantitative IE would also be explored. The mode of design and delivery would involve a core partnership between PSF, PREM and identified local partners (currently discussing with Akatiga and SMERU), and may involve potential international partners in a related practice network (see para. 23 below), once endorsed by the JMC. ii. Organisational strengthening. While the Asia Foundation has trialed a holistic organizational approach to strengthening of local CSOs (under the Knowledge Sector pilot), the PSF M&E team does not have a comparative advantage in overall organizational capacity review/support, so, under this proposal, is planning an approach more in keeping with the team s technical skills and needs. This would involve targeting selected local research institutes with undeveloped potential (the so-called 2 nd tier of independent research institutes), particularly in qualitative methodologies, through potential developmental assignments of selected individual staff within PSF over a month period. Current thinking would be to mobilize a team of individuals selected from a number of local institutes, to come on assignment to PSF and participate as members of a dedicated qualitative support team for an identified period, prior to returning to their parent institutions with a commitment to then remain with those parent institutions for a defined period. iii. Enhanced partnership model for discrete studies (e.g., Papua UNCEN/UNIPA model). Taking into account a study s geographic or thematic focus, the team will engage related university bodies/faculty and other CSO representatives to sit on a study oversight/review panel as Senior Advisors. These key faculty will contribute to concept design, participate in methodological training exercises alongside PSF team members and the hired local research institution; and assist in recruiting a number of their students on learning field assignments as Junior Researchers alongside teams in the field. These Junior Researchers would have discrete outputs that would not directly impact the overall study output, but would introduce the students to the processes of data gathering, analysis and presentation of findings. They would be mentored by the Senior Advisors and exposed, through the process, to methodological training; the practice and experience of working with a professional (local) research body and interaction with the PSF team 23. Future partnerships and innovations in delivery: In parallel to delivery of this key phase of evaluations, the team is exploring options for developing additional critical partnerships, with both local and international partners, to build networks for a collaborative research and capacity building program in Indonesia, taking advantage of entry points offered by the Knowledge Sector initiative and with a focus on strengthening of qualitative and applied ethnographic methodologies 3. The team may also explore the potential for a contractual partnership with an experienced firm who may take on leadership and delivery of a discrete package of the existing work covered in this proposal, with guidance and quality assurance from the M&E team. 3 A separate Note is available describing these plans. Page 14 of 25

15 VI. Estimated Budget 24. The Project team is currently seeking approval of US$4,750,000 for the two-year project (See below for a breakdown and Annex 3 for a breakdown of specific evaluation/study budgets). Category (1) Evaluations/studies $3,000,000 (2) Capacity building for local research institutes $350,000 (3) Supervision and implementation support including staffing, field oversight $1,400,000 ALLOCATION REQUESTED $4,750,000 US$ Page 15 of 25

16 Annex 1: PSF M&E Program, Studies Completed or Ongoing in Table 1: Completed Evaluations/Studies Evaluations/Studies Methodology Main Findings 1 PNPM-Rural Impact Evaluation Quantitative (propensity score matching); endline 2010 Real per capita consumption gains of 9.1 % for households in PNPM-Rural areas The gains were higher for poor households and poor sub-districts (11.8% and 12.7% respectively) Households in PNPM areas were 2.1% more likely to move out of poverty Access to health services increased 5.1% for households in PNPM areas 2 PNPM-Rural Qualitative Impact Evaluation Qualitative, multi-years (2007 and 2010) Participation, transparency and accountability were strong within the program, especially at the sub-district and village levels PNPM is most effective at reducing poverty and impacting poor households when the needs of the poor are aligned with those of the community 3 PNPM-Rural Marginalized Groups Study Qualitative (2010) Marginalized groups have limited participation in PNPM meetings which were still dominated by elites and interest groups Although facilitation/project procedures have often led to increased participation rates, it has not encouraged active or influential participation of marginalized groups in the development of proposals Officials and leaders of interest groups (religious and traditional elites) still possess the greatest influence over which proposals are developed and selected, therefore reducing opportunities for marginalized groups to impact Page 16 of 25

17 Evaluations/Studies Methodology Main Findings decision-making on the use of project resources 4 Village Capacity in Maintaining Infrastructure Study: Evidence from Rural Indonesia Quantitative (repeated surveys every quarter in a year to capture cyclical income fluctuation, 2010) Many communities still found it difficult to maintain existing infrastructure projects The cost of maintaining infrastructure was up to 2.8 percent of a household s total consumption although small is likely burdensome for poor households Although villagers are willing to pay for the maintenance of infrastructure, the amount communities are willing to pay does not cover all infrastructure maintenance needs The community s willingness to pay is strongly influenced by the direct impact of the infrastructure on individual households and institutional responsiveness in terms of complaint handling 5 PNPM Generasi Impact Evaluation Quantitative (randomized control trial, baseline 2007, endline 2010) Significant impacts on all 12 indicators The strongest improvements among the health indicators were in the frequency of weight checks for young children The improvement in education indicators was most notable in the increased school participation rate among the primary school-age group In terms of long-term impact, there was about 10 percent decrease in malnutrition compared with control areas 6 PNPM RESPEK Infrastructure and Community Organization Capacity Evaluation Mixed-methods (2011) All infrastructure samples evaluated are of good to moderate quality significant achievement considering huge implementation challenges in Papua and West Papua 67% of the infrastructure built were utilized by the community Page 17 of 25

18 Evaluations/Studies Methodology Main Findings PNPM RESPEK improved administrative capacity of local facilitators but not the facilitation capability 7 Rapid Assessment of Women s Participation in PNPM Qualitative (2012) Quality of women s participation was mixed in study areas and is still not maximized Women s involvement in the project cycle is still low beyond the initial project stages of socialization and needs assessment Women s proposals that are funded are usually infrastructure, rather than e.g., capacity building and trainings that are requested when the local actors do a more in-depth analysis of women s needs Local initiatives and strategies around women s participation existed but yet to be integrated into more general local decision making processes 8 Local Level Governance Review Qualitative/ Action Research (2012) The state of local governance in PNPM is mixed: it is still remarkably strong, but faces problems. Participation rates are still mostly high, but the quality of participation has declined in some places. There are weaknesses in transparency and information sharing: performance of accountability mechanisms in the program varies and the incidence of serious corruption is up. Despite this, the foundations of PNPM are still strong. These foundations, though, are being eroded by serious pressure from higher-up problems related to the broader governance environment, changes in the program design, and problems with implementation and management. PNPM must address these program design, implementation and management issues to avoid undoing its years of good work at community level. 9 Rate of Return Analysis (EIRR) of PNPM-Rural Quantitative (economic analysis), small Similar with 2005 results (Torrens, 2005), the EIRR varied among subprojects with median value of 30 50% The average general income multiplier is 1.3 Page 18 of 25

19 Evaluations/Studies Methodology Main Findings Infrastructure Sub-Projects scale study in 20 villages assessing 48 sub-projects using similar methodology as in 2005 exercise Sub-projects are generally 25 30% cheaper than project built using typical local government contractor 10 Village Infrastructure Census Quantitative (village level census) piggybacking PODES 2011 Comprehensive data on basic infrastructure availability and quality (including health and education facilities) in nearly all urban and rural villages in Indonesia (over 76,000 villages). A consistent picture of geographic variation in the supply of basic infrastructure across Indonesia. In general, the island of Java and the province of Bali perform best, while local needs for investment still exist in these regions, and particularly so in the provinces of Jawa Barat and Banten. The largest gaps in infrastructure supply readiness are found for the Papua region, the Maluku islands, NTT, as well as for the interior of Kalimantan. Data has been used to create a supply readiness index for health and education (to support PNPM Generasi in location selection/targeting). Further analysis will be done to calculate financing gaps (pending data availability on costs) and to provide more information for PNPM-Rural in terms of location selection and allocation of block-grants Table 2: Ongoing Evaluations/Studies: 2012 No Name of Study Description of Study Methodology Expected Timeline for Result 1 Local Level Observes changes in local organizational capacity and social capital (embodied in participation in community- Mixed-methods (longitudinal study, First draft report Page 19 of 25

20 No Name of Study Description of Study Methodology Expected Timeline for Result Institutions 3 (LLI3) based organizations); links changes in such capacity to shifts in community-members influence over development decision-making, project implementation, and service provision at the district and community levels. 1 st one in 1996 and the 2 nd in 2001) expected Q Incidence of Household Benefits Determining who actually benefits from PNPM as well as understanding community s perspective of PNPM including its uses and its capacity to target the poor and vulnerable in particular. Quantitative Report expected Q Integration of Community-based Poverty Reduction Programs at Community Level Examining integration of CDD programs at community level and observing local community capacity to manage diverse development programs; lessons learned will be used as inputs to the development of integration strategy in PNPM Roadmap. Qualitative First draft report expected Q Papua Operational Research and Evaluation Several analytic pieces to be done related to PNPM RESPEK and conflict issues in Papua: (i) a qualitative beneficiary assessment to elevate voices from the field; (ii) piggybacking SUSENAS with a specific PNPM module to collect and benchmark quantitative data/indicators from Papua and West Papua; and (iii) operational mapping/location profiling in Papua. Mixed-methods Concept note(s) completed by Q Page 20 of 25

21 Annex 2: Mapping of PSF Studies to Organising/Research questions. Research Question Policy/program relevance Ongoing or Proposed Activity 1. Continued poverty impact: How, why and where does PNPM reduce poverty? a. How are local economies, productivity and livelihoods affected by PNPM infrastructure? In what contexts does PNPM work better? Should the program be geographically targeted? How to adapt PNPM for further livelihoods development? i. PNPM Evaluation (Econ Impact Simulation) Anticipated timeline i b. Is PNPM as an instrument fulfilling its stated social and economic objectives? What is the role of Cluster 2, vis a vis Clusters 1 and 3? c. What is PNPM s effectiveness in Indonesia s most disadvantaged/poores t regions? (see also Q2.) (e.g., evaluation shows greatest poverty impact in poorer areas/households, but implementation challenges and bottlenecks are greatest in poorest and most remote areas) Crystallising PNPM s role and efficacy in the logic of GoI s poverty strategy, mediating between C1 and C3. Optimising PNPM effectiveness where needs and challenges are greatest. Should PNPM design, activities and indicators be adjusted for regional/sociocultural variation? i. PNPM Evaluation (Econ Impact Simulation and possible 2013 Sedap repeat round) ii. Sentinel Villages iii. RLF Pilot Evaluation (tbc) iv. Eval of combined effects of PKH and Generasi in areas of overlap i. Papua PNPM Rural/Respek Qualitative Voices study ii. Papua Rural/Respek Quantitative Evaluation (SUSENAS 2013) iii. PNPM Evaluation i ii iii. Tbc iv. tbc i ii iii d. Which aspects or features of PNPM are performing better? How does the PNPM Identifying the most effective parts of the PNPM process/delivery mechanism. Continued justification of i. Rural infrastructure census ii. PNPM i ii iii Page 21 of 25