World Vision s Approach to Local Ownership in Evaluation. Korça, Albania Development Program a case study

Transcription

1 World Vision s Approach to Local Ownership in Evaluation Korça, Albania Development Program a case study Prepared by: Holta Trandafili and Albana Dino, World Vision US March 2015

2 Summary Intro to the Development Program Approach (DPA) and how that informs our evaluation process Background of Korça, Albania Development Program Role of local actors in the evaluation process Why invest in local ownership in evaluations When to use this approach to evaluations What does it take? Who and timing? Testimonials 2

3 Development Program Approach Sustained well-being of children within families and communities especially the most vulnerable Includes: Understanding of partnering Types of partners The critical path Includes: Child well-being outcomes Child participation Defined project models Measurements Working with partners Contributing towards child well-being Includes: Roles of WV local -level staff Key competencies Management support Equipping local-level staff Basic programme parameters Includes: Geographic size Target populations Life span and cycle Resourcing Governance Disaster management Sponsorship programming (where applicable) 3

4 The Critical Path for Working with Communities and Local Partners Baseline data collection & analysis Programme/Project Descriptions, incl. Indicators (All Partners) World Vision Logframe and Design Document Assessment Plan Assessment Report

5 Korça Area Development Program (ADP) Program Phase Evaluation Type Evaluation Purpose Primary Methodologies Implementer Funder 1 October September 2013 (1st implementation phase) Mid-term Programme Evaluation (end of 1st implementation phase); Formative; To generate learning from the program s first cycle design and operation to inform the redesign process and product. Evaluation methodology was based on participatory approaches, based on naturalistic inquiry (qualitative); Details were developed by the Lead Evaluator World Vision Albania World Vision Hong-Kong Evaluator Intended users of Evaluation Product External evaluator hired by World Vision International Korça ADP local partners and World Vision Albania (redesign for next phase) World Vision Hong Kong (inform donors on progress) World Vision International (organizational learning) Duration 4 months (February May 2013) Program target population Projects in the program The ADP area has 50,100 inhabitants (8,432 children aged 0-18 years old) and operates in three rural areas and one small municipality (Pojan, Vreshtas, Libonik, and Maliq) registered children in Sponsorship project (aged 1-11) Two projects implemented: Education and Sponsorship

6 Who were these local actors in involved in Korça Program Evaluation? 48 local partners with solid foundational relationships established from program inceptions and who are still actively sustained were initially contacted; 20 of them participated in workshops and 10 of them were selected by the network to represent the local actors voice through-out the process. The types of partners range from informal groups to established CBOs, FBOs and local NGOs or local government and regional directorates.

7 The nature of the participatory approach of Korça evaluation What it was NOT: A capacity building process that trains community members to collect information for World Vision Simply a process to get information from the community for World Vision reporting needs A one time off participatory event (actors / partners called on purpose for only this step evaluation) What it was: A capacity development and learning process that provides opportunity for all actors to meaningfully participate in all the steps of Evaluation. A process that seeks to meet the needs of all partners (existing community groups, individual volunteers, local government reps, local NGOs, etc.,) A continuation of working in partnership to jointly design, baseline, monitor and measure the program progress.

8 How were participants involved? Overview of process Local staff and partners agree to commence the evaluation process Share draft ToR with local partners Local partners shape ToR (link Theory of Change, purpose of Eval and Objectives) Agreed ToR for review of National Office, Funding Office and WVI Recruitment of the Evaluation Leader to deliver on the ToR Articulating biases of involving local partners (informant vs. part of eval. team) Articulating evaluation limitations (what it can say and what not) Develop sampling framework Mapping local social activities, interventions, beneficiaries. Finalize indicators (articulation of success in local terminology) Evaluation Design Workshop (link Theory of Change, purpose of Eval and Objectives) Chart partners in informants vs. eval. team participants. Local eval. team members determine level of engagement and in what step(s). Finalize tools Build capacity of eval. Team members on tools. Form data collection teams Determine follow-up and next steps Validation of findings Presentation of findings, reflection & recommendations (from Eval. Team to network and wider) Data analysis Data Collection Report drafting (different formats) Final report shared with Funding Office Legend Community involvement WV facilitates/ builds capacity WV process

9 Topics covered by the Evaluation Questionnaire Impact Partnership Partner Learning and Capacity Sponsorship Most Vulnerable Sustainability Future Field Work 17 Focus Groups 12 Key Informant Interviews 5 Korça ADP Staff Interviews 3 Partner Interviews

10 Initial Interpretation Step One: Type up focus group and key informant interview responses 5 teams 70+ pages of notes Step Two: Describe the data Step Three: Present findings Step Four: Formulate Recommendations

11 Participatory Evaluation Process

12 Group Work -Analysis Evaluation Findings

13 Evaluation Discussion & Debate

14 EVALUATION Celebration

15 Evaluation Team

16 Participants Evaluation of the Locally Owned Evaluation in Korca, Albania Positive Partners included Discussion and debate Feeling of equality partners, ADP staff, national office Qualitative evaluation process modeled and included full participation at every step External party facilitating evaluation Work in groups Engagement and commitment of WV staff Translation

17 Participants Evaluation (2) Needs Improvement Not enough time for discussion and debate No accreditation or certification for participants who are not staff members Not enough time for data collection and analysis Broader sample of focus groups required More in depth and better review of sponsorship project required More review between evaluation teams and lead evaluator (more direction/feedback required) More / deeper questions required of interviewees

18 Why use Local Ownership in Evaluations? It puts information in the hands of the main user: the community! It can save money

19 Local Ownership to Evaluations when to use it? Enabling factors: Substantive in-depth time invested up front for the initial community engagement process Acknowledgment and joining paths with community actors their vision, plans, articulation of success indicators Staff commitment to Community Empowerment Approach (local ownership, meaningful participation, capacity development) Rules of engagement of effective partnership are continually adhered to: a) Concern for equity, b) Commitment to transparency, and c) Emphasis on mutual benefit 19

20 Challenges to consider: If local ownership and participation was not build in the design phase then applying local ownership in evaluation is difficult. Difficult to apply in communities with little to no community organization The involvement of technical specialists, while necessary in an evaluation requires their buy-in a participatory process and working with the groups to present the process and the findings in palatable way (layman terms and using local language). The big assumption that: we have the capacity to lead these highly participatory processes and the organization is ready for meaningful shared learning and sound recommendations Challenges with time... 20

21 Summary A shared project/program suggests shared ownership and in the context of a shared ownership we can have locally owned evaluations. Locally owned evaluations are not a coincidence they are the product of hard work to build ownership over the project during the whole program/project cycle. Local ownership is the highest level of participation where owners are conscientious about what is happening... And they can shape the process and implement it and above all they learn from it. The level of conscience of the local groups / partners should lead to understanding where they are with the competences and when to seek assistance from technical specialist to guide for rigor 21

22 Korça Testimonials Partners unanimously praised World Vision Korça ADP for their outstanding commitment to genuine partnership. One partner said, I have worked with other organizations in other places, and in these other places, I have felt that when World Vision talked to us about partnership, they were really trying to see how much milk they could get out of the cow. I have not felt that way at all in Korça, and have profited immensely from the relationship with World Vision. They truly have put everyone on an equal plane and are trying to serve others, not just themselves and their plans.

23 END