Comprehensive Evaluation of KOICA WFK Program

Transcription

1 Comprehensive Evaluation of KOICA WFK Program

2

3 The Korea International Cooperation Agency (KOICA) performs various types of evaluation in order to secure accountability and achieve better development results by learning. KOICA conducts evaluations within different phases of projects and programs, such as ex-ante evaluations, interim evaluations, end-ofproject evaluations, and ex-post evaluations. Moreover, sector evaluations, country program evaluations, thematic evaluations, and modality evaluations are also performed. In order to ensure the independence of evaluation contents and results, a large amount of evaluation work is carried out by external evaluators. Also, the Evaluation Office directly reports evaluation results to the President of KOICA. KOICA has a feedback system under which planning and project operation departments take evaluation findings into account in programming and implementation. Evaluation reports are widely disseminated to staffs and management within KOICA, as well as to stakeholders both in Korea and partner countries. All evaluation reports published by KOICA are posted on the KOICA website. ( This evaluation was entrusted to Re-shaping Development Institute by KOICA for the purpose of independent evaluation research. The views expressed in this report do not necessarily reflect KOICA's position.

4 AusAID AVID CAN CIS CPS DFID FGI ICS ICT ILO JICA JOCV KCOC KOICA KOVA LFS LMICs MDGs NCS NGO ODA OECD/DAC OJT SDGs UNDP UNV VISES VSO WFK Australian Agency for International Development Australian Volunteers for International Development Community Need Assessment Commonwealth of Independent States Country Partnership Strategy Department for International Development(DFID) Focus Group Interview International Citizen Service (UK) Information and Communication Technology International Labor Organization Japan International Cooperation Agency (JICA) Japan Overseas Cooperation Volunteers Korea NGO Council for Overseas Development Cooperation Korea International Cooperation Agency Korea Overseas Volunteers Association Labor Force Survey Lower Middle Income Countries Millenium Development Goals National Competency Standards Non-governmental Organization Official Development Assistance Organization for Economic Cooperation and Development/ Development Assistance Committee On the Job Training Sustainable Development Goals United Nations Development Programme United Nations Volunteers Volunteer-Initiated Self-Evaluation System Voluntary Service Overseas World Friends Korea

5 Chapter 1. Evaluation Overview 1 1. Evaluation Background and Goals 3 2. Evaluation Target, Scope and Criteria 5 3. Evaluation Framework 8 4. Evaluation Methodology Limitations of the Evaluation 21 Chapter 2. Theoretical Background of the Evaluation Trends in Evaluating International Volunteer Program Evaluation Methodology for International Volunteer Program 28 Chapter 3. Implementation Situation of KOICA WFK Strategy Implementation Situation 42 Chapter 4. Examination of KOICA WFK Results Volunteers Hosting Country (Improved Standard of Living) Sending Country (Strengthening Ties with Hosting Country) 78 Chapter 5. Implementation Plan for Results-Based Management KOICA WFK Improvement Plans KOICA WFK System for Managing Results 89 Chapter 6. Conclusion and Recommendations Conclusion Recommendations 96

6 [Tables] [Table 1-1] Evaluation Criteria According to Evaluation Targets 7 [Table 1-2] Strategy and Processes Evaluation Framework 8 [Table 1-3] Categories of Volunteer Results Used in the Evaluation 12 [Table 1-4] Results Evaluation Framework: Output 13 [Table 1-5] Results evaluation Framework: Outcomes and Goals 16 [Table 1-6] Summary of Evaluation Method 20 [Table 2-1] Results of International Volunteer Project: Recipient country, Volunteers, Donor country 27 [Table 2-2] Main Evaluation Methods of International Aid Agencies 30 [Table 3-1] WFK Vision 35 [Table 3-2] Adequacy of Recruitment and Hiring Procedures 48 [Table 3-3] Adequacy of Post-Project Management 55 [Table 3-4] KOICA WFK Evaluation 57 [Table 4-1] Results per Level: Volunteer 64 [Table 4-2] Survey Results: Volunteers' Perception of Capacity (Building) 65 [Table 4-3] Survey Results: Volunteers' Perception of Character Cultivation 66 [Table 4-4] Survey Results: Volunteers' Perception of Global Networks 67 [Table 4-5] Survey Results: Volunteers' Perception of Results at Outcome Level 68 [Table 4-6] Differences in Perception of Results According to Perception of Implementation Systems: Overall 69 [Table 4-7] Results per Level: Hosting Country 71 [Table 4-8] Result Indicators for Outcomes from Hosting County 72 [Table 4-9] Number of educational training sessions and number of participants 74 [Table 4-10] Survey Results: Cooperating agency personnel on results at the Outcome level 75

7 [Table 4-11] Survey Results: Volunteers on results at the Outcome level 76 [Table 4-12] Differences in Perception of Results According to Perception of Implementation Systems: Overall 77 [Table 4-13] Results per Level: Sending Country 79 [Table 4-14] Enhance relationships between hosting and sending country: Cooperating agency 80 [Table 4-15] Differences in Perception of Results According to Perception of Implementation Systems: Overall 81 [Table 5-1] Summary of ways to improve results for volunteers: Contribute to the nurturing of global talent 87 [Table 5-2] Summary of ways to improve results for volunteers: Improving living standards 88 [Table 5-3] Summary of ways to improve results from hosting country: Improving living standards: Improved relationships between hosting and sending country 89

8 [Figure] [Figure 1-1] Evaluation Background and Scope 6 [Figure 1-2] Theory of Change for the Evaluation 12 [Figure 3-1] KOICA WFK Implementation System (Organization Chart): Apr [Figure 3-2] KOICA WFK Implementation Steps 44 [Figure 3-3] Important factor for fulfilling demand by cooperating agencies 48 [Figure 3-4] Adequacy of Recruitment and Hiring Procedures 48 [Figure 3-5] Requirements for systemization of volunteer recruitment and selection 50 [Figure 3-6] Improvements required in for training 51 [Figure 3-7] Recommendations for post-project management 56 [Figure 4-1] Theory of Change: Volunteers 63 [Figure 4-2] Theory of Change: Hosting Country 70 [Figure 4-3] Percentage of Types of Local Project Support ( ) 73 [Figure 4-4] Percentage of Areas of Local Project Support ( ) 73 [Figure 4-5] Theory of Change: Sending Country 78

9 Ⅰ. Evaluation Overview 1. Evaluation Background and Goals 2. Evaluation Target, Scope and Criteria 3. Evaluation Framework 4. Evaluation Methodology 5. Limitations of the Evaluation

10

11 Chapter I Evaluation Overview 1. Evaluation Background and Goals Recently the international community has emphasized the importance of tangible results for development and implementing results-based project management to improve aid effectiveness. Volunteer projects are seen as the most suitable way to implement the Sustainable Development Goals (SDGs). Furthermore, the importance of using both qualitative and quantitative indicators is becoming important in evaluating the impact that volunteer activities have on development results. In light of these discussions within the international community, KOICA is creating a results-based management indicator pool for the KOICA WFK (World Friends Korea) Project. Until recently, the evaluations that were conducted were limited to asking for satisfaction levels from individual volunteers and the institutions in host countries. Previously there have been no evaluation indicators or methodology, so it has been hard to evaluate whether the mid-to-long term goals of the project have been reached or not. More specifically, the yearly evaluations conducted by the international offices have specific evaluation indicators in each evaluation category, and the comprehensive evaluation conducted by the headquarters lacks in the provision of the lessons learned and recommendations for overall management and the establishment of a results-based management system (RBM). The satisfaction level survey has also been limited to the individual volunteers that participated that year, and thus it Evaluation Overview 3

12 was hard to evaluate whether the distribution country had reached its goals or not. Therefore, this comprehensive evaluation aims to assist in overcoming the limitations of the current evaluation practices and diagnose the volunteer project's current management situation. It also aims to evaluate the results of KOICA's International Volunteers and create an effective results-based management system that is based on the evaluation. The total number of KOICA volunteers that were dispatched between stands at 11,007 volunteers to 65 countries. Between an average of 1,000 volunteers from KOICA WFK was dispatched every year - the number of volunteers has been increasing by a large number every year. Therefore, it is important to evaluate the management situation and results of the KOICA WFK, so that a consensus can be reached from a results-based management perspective. Currently the evaluation of the KOICA WFK is done through annual comprehensive evaluations and satisfaction level surveys. However, after reviewing KOICA WFK s evaluation report it was found to be limited to satisfaction level surveys for individual donors and institutions from receiving countries. Therefore it is difficult to draw any recommendations for improvement or to evaluate the mid-to-long term goals of the KOICA WFK. Considering this, the necessity of this evaluation can be categorized into internal and external aspects. External Aspect: The results of the KOICA WFK has become increasingly important, so there is a need to structure the results in a more logical and systematic way and to specify the program. Internal Aspect: The importance of having implementation and results-based 4 Comprehensive Evaluation of KOICA WFK Program

13 management systems that are based on the characteristics of the KOICA WFK has also become important. Therefore, recommendations for establishing such systems, based on actual evaluations are necessary. In conclusion, this evaluation report aims to comprehensively evaluate the situation and results of the KOICA WFK over the past 3 years and highlight how the project has contributed to development goals. In doing that, it will contribute to the overall improvement of the implementation system and results-based management system. 2. Evaluation Target, Scope and Criteria The target of this evaluation is the overseas development volunteer project that KOICA implements (WFK KOICA Volunteer Team). The evaluation is separated into two sections - firstly, an analysis of the KOICA WFK implementation situation and the second, the KOICA WFK results from Cambodia and Rwanda. The reason these two countries were chosen was because 1) The regions where WFK KOICA volunteers were most dispatched between was Asia (55.8%) and Africa (22.1%), therefore choosing Cambodia and Rwanda represents both regions. 2) Despite the fact that Cambodia and Rwanda had volunteers come for the first time recently (2003 and 2006 respectively), they rank within the Top 10 countries for highest number of volunteers. In 2013, there were 96 volunteers dispatched to Cambodia, which makes up for 6.3% of the total 1,163 volunteers dispatched. In Rwanda, there were 44 volunteers dispatched (3.8%). It was determined that these two countries could provide specific information on the actual implementation situation of the WFK. The scope of the evaluation was divided into three parts <Part 1. Situational Evaluation Overview 5

14 Analysis of the KOICA WFK>, <Part 2. Inspection of KOICA WFK Results>, <Part 3. KOICA WFK Results-Based Management Implementation Methods (which is based on Part 1 and Part 2)>. In Part 1, an evaluation matrix was created to analyze the KOICA WFK situation (project strategy, progress and implementation system). In Part 2, the theory of change model and evaluation matrix was used to determine the WFK's main results and lessons for future recommendations, based on the project results. In Part 3, to improve project results, recommendations for the implementation system were made. In particular recommendations are made for how to manage results in order to implement results-based management in KOICA WFKs. [Figure 1-1] Evaluation Background and Scope 6 Comprehensive Evaluation of KOICA WFK Program

15 The evaluation was conducted while adhering to the OECD Development Assistance Committee's (DAC) Criteria for Evaluating Development Assistance - Relevance, Effectiveness, Efficiency, Impact and Sustainability. Under this there are three categories - strategy, process and results, and the evaluation criteria which are applied differently to each criterion. [Table 1-1] Evaluation Criteria According to Evaluation Targets Strategy Process Results Target KOICA WFK Strategy - WFK Comprehensive Project Plan ( ) KOICA WFK Department Strategies (14-17) - Respective country strategies for Asia, Africa, Latin America, Eastern Europe CIS KOICA WFK Implementation System - Demand Analysis, Recruitment and Hiring, Training, Dispatch (management of conduct and activities), post-project Management Results achieved in the Cambodia WFK Results achieved in the Rwanda WFK Criteria Relevance Relevance, Efficiency Effectiveness, Impact, Sustainability Evaluation Overview 7

16 3. Evaluation Framework The KOICA WFK evaluation team used an evaluation framework consisting of: strategy evaluation, processes evaluation and results evaluation, in order to evaluate the implementation situation and results of each country. 1. Analytical Framework for Evaluation of Project Implementation: Strategy and Processes Evaluation Framework To evaluate the project's implementation state, the team developed an evaluation criteria and questionnaire to evaluate the project's relevance and efficiency. This was based on the strategy and processes evaluation framework. Furthermore, to analyze the actual relevance and efficiency of the project's implementation system, specific evaluation questions were created. The questions were categorized according to each step in the implementation process (implementation system, demand analysis, hiring and recruitment, training, activity support, post-project management, monitoring and evaluation). [Table 1-2] Strategy and Processes Evaluation Framework Category Criteria Target Main Evaluation Questions Strategy Relevance Relevance of the WFK's strategy Are the elements within the strategy (goal, direction, implementation strategy, allocation of resources) logical and systematic? Do the project personnel and participants share a common strategic goal and direction? Is there a sufficient institutional foundation to reach the project's goals? Do the goals and strategy align with the government's Fostering Young Global Leader policy? Methodology and Source Methodology: Literature Analysis, Interviews with Korean personnel Source: WFK strategy documents, Government strategy documents, Interview results 8 Comprehensive Evaluation of KOICA WFK Program

17 Category Criteria Target Main Evaluation Questions Strategy Processes Relevance Relevance Efficiency Alignment between the WFK's strategy and Korea's ODA policies Alignment between WFK strategy and the host country's demands Alignment between WFK's strategy and development aid norms WFK implementation system Are the hosting countries Priority Partner Countries? Are the hosting countries low income countries? Does the project align with KOICA's strategy and focus areas? Does the WFK country strategy align with the host country's demands? Does the WFK country strategy align with the respective host country's policies? Do the WFK respective country strategies make predictions for the host country's demands? Does the WFK strategy consider international development aid norm (SDGs and etc.) and the strategic direction of the international community/other international volunteer organizations? Does the WFK strategy have a way to adapt to future changes in development aid norms? Is the decision-making process and division of labor within WFK headquarters appropriate? Does KOICA have an appropriate role within the project's comprehensive implementation strategy? Is there effective cooperation between headquarters and the implementing agency, and headquarters and the local office? Does the project make use of the Korean and international network in order to bolster professional aspects of the WFK? Methodology and Source Methodology: Literature Analysis, Interviews with Korean personnel Source: WFK and KOICA strategy documents, Interview results Methodology: Literature Analysis, Interviews with Korean and foreign personnel Source: WFK strategy documents, hosting country's development policy documents, Interview results Methodology: Literature Analysis, Interviews with Korean Personnel Source: WFK strategy documents, Documents on international development aid norms, Interview results Methodology: Interviews with Korean personnel Source: Interview results Evaluation Overview 9

18 Category Criteria Target Main Evaluation Questions Processes Relevance Efficiency Demand analysis Recruitment and hiring Training in Korea and host country Activity support Is there a process to analyze the host country's demand? Is there an appropriate process to verify the host country's demand? Is there alignment between the expertise of the volunteers and the host country's demand? Is the recruitment and hiring process efficient? Is the recruitment and hiring process fair? Are the period, amount and contents of the training in Korea appropriate? Are the period, amount and contents of the training in hosting countries appropriate? Is there an appropriate safety management system for volunteers after their dispatch and while they are working? Is there an appropriate administrative support system (for supplies and local project support) for volunteers? Are the local community and cooperating agencies supportive of the volunteer activities? Methodology and Source Methodology: Literature Analysis, Interviews with Korean personnel Source: Information from host country stating their demand, Interview results Methodology: Literature Analysis, Interviews with Korean personnel Source: Documents on hiring and recruitment Interview results Methodology: Literature Analysis, Interviews with Korean Personnel Source: Training material for volunteers, Interview results Methodology: Literature Analysis, Interview with Korean personnel Source: Material on activity support for the volunteers, Interview results 10 Comprehensive Evaluation of KOICA WFK Program

19 Category Criteria Target Main Evaluation Questions Processes Relevance Efficiency Post-project management Monitoring and evaluation Is there an appropriate post-project management system (support type, nature of the support)? Has there been an appropriate evaluation of the volunteer activities? Is there an appropriate data management system for volunteer activities? Methodology and Source Methodology: Literature Analysis, Interview with Korean personnel Source: post-project management material, Interview results Methodology: Literature Analysis, Interview with Korean personnel Source: Evaluation material on volunteers Interview results 2. Analytical Framework for Evaluation of Project Results: Theory of Change and Results Evaluation Framework For this evaluation a theory of change was created, as well as a results-evaluation framework based on social capital theory. These act as the framework to evaluate the project results. The significance of this evaluation lies in the fact that it focuses on elements critical to achieving project results and concepts that are hard to quantify. Networking, people-centered approaches, partnerships, motivations beyond economic value, openness to ideas and information exchange are the main elements of social capital. The idea of social capital was expanded to five different types (material capital, economic capital, human capital, social capital, cultural capital) to create a model that categorizes the project results. The evaluation team took these five categories and worked them into three categories to make it appropriate for the KOICA WFK's theory of change. Evaluation Overview 11

20 [Table 1-3] Categories of Volunteer Results Used in the Evaluation Category Definition Example Material and Economic Human Social and Cultural Economic benefit of value increase on the public goods market. Knowledge, skills and abilities beneficial to economic activities. Special types of resources such as networks, trust and norms that facilitate social exchange. Artistic abilities and cultural tastes of individuals, families and other members of society. Tangible cultural products. Number of installed computers Increase in income Employment rate Certifications Participation rate of NGOs Internet usage rate The theory of change model that shows the logical connections among the: Activities, Output, Outcomes and Goal of the KOICA WFK was created. Then, the Goal, Outcome, Output were broken down to create a results evaluation framework that evaluates the efficiency, impact and sustainability of the project. [Figure 1-2] Theory of Change for the Evaluation 12 Comprehensive Evaluation of KOICA WFK Program

21 [Table 1-4] Results Evaluation Framework: Output Category Criteria Target Aspect Main Evaluation Questions Results Efficiency Outputs Achieved Volunteer Aspect (Capacity building) Have your foreign language skills improved through volunteer work? Has volunteer work contributed to your (professional) work experience? Have you improved your leadership skills? Have you improved your problem-solving skills? Have you increased your understanding of the host region and country? (Cultivation of Character) Have you become more open-minded through volunteer work? Have you become more independent through volunteer work? Have you increased your self-confidence through volunteer work? (Establishing a global network) Have you established a network consisting of locals through the volunteer work? Have you established a network with foreign volunteers from other international aid agencies? Have you established a network among WFK volunteers? Methodology and Source Target: Volunteers Method: Interviews, Questionnaires Source: Interview results, Questionnaire results Evaluation Overview 13

22 Category Criteria Target Aspect Main Evaluation Questions Results Efficiency Outputs Achieved Host Country (Public services) Has the provision of human resource services increased through the work of volunteers? Have the provision of material services (facilities, equipment) increased through the work of volunteers? (Improved capacity for hosting agencies) Has there been an increase in training contents for capacity building? Has there been an increase in training programs for capacity building? Has there been an increase in hosting agency personnel participating in the training? (Improved policies for host agencies) Has the administrative system used by host agencies for volunteer work improved or been developed? Has there been an increase in policies and research opportunities to improve volunteer work? Methodology and Source Target: Volunteers, Local cooperating agencies Method: Literature Analysis, Interviews, Questionnaires Sources: Information on the results/situation of the volunteer team, Interview results, Questionnaire results 14 Comprehensive Evaluation of KOICA WFK Program

23 Category Criteria Target Aspect Main Evaluation Questions Results Efficiency Outputs Achieved Distributing Country (Increase in regional experts) Has the expertise on the region and country improved? Has there been increase in the number of volunteers who have completed their volunteer work? (Perception of distributing country and agency) Has the understanding of KOICA activities increased through volunteer work? Has the sustainability of KOICA work increased through volunteer work? Has the perceived image of Korea improved through volunteer work? Methodology and Source Target: Local cooperating agencies, KOICA HQ Method: Interviews, Questionnaires Source: Interview results, Questionnaire results Evaluation Overview 15

24 [Table 1-5] Results evaluation Framework: Outcomes and Goals Category Criteria Target Aspect Main Evaluation Questions Results Effectiven ess Achievement of mid-term results Volunteers Aspect (Capacity building) Have your foreign language skills improved through volunteer work? Has volunteer work contributed to your (professional) work experience? Have you improved your leadership skill? Have your improved your problem-solving skills? Have you increased your understanding of the host region and country? (Cultivation of Character) Have you become more open-minded through volunteer work? Have you become more independent through volunteer work? Have you increased your self-confidence through volunteer work? (Establishing a global network) Have you established a network consisting of locals through volunteer work? Have you established a network with foreign volunteers from other international aid agencies?? Have you established a network among WFK volunteers? Methodology and Source Target: Volunteers Method: Interviews, Questionnaires Source: Interview results, Questionnaire results 16 Comprehensive Evaluation of KOICA WFK Program

25 Category Criteria Target Aspect Main Evaluation Questions Results Effectiven ess Achievement of mid-term results Host Country Distributing Country (Public services) Has the provision of human resource services increased through the work of volunteers? Has the provision of material services (facilities, equipment) increased through the work of volunteers? (Improved capacity for hosting agencies) Has there been an increase in training contents for capacity building? Has there been an increase in training programs for capacity building? Has there been an increase in host agency personnel participating in the training? (Improved policies for hosting agencies) Has the administrative system used by host agencies for volunteer work improved or been developed? Has there been an increase in policies and research opportunities to improve volunteer work? (Increase in regional experts) Has the expertise on the region and country improved? Has there been an increase in the number of volunteers who have completed their volunteer work? Methodology and Source Target: Volunteers, Local cooperating agencies Method: Literature Analysis, Interviews, Questionnaires Sources: Information on the results/situation of the volunteer team, Interview results, Questionnaire results Target: Local cooperating agencies, KOICA HQ Method: Interviews, Questionnaires Source: Interview results, Questionnaire results Evaluation Overview 17

26 Category Criteria Target Aspect Main Evaluation Questions Results Effectiveness Achievement of mid-term results Impact and sustainability Achievement of long-term results Distributing Country Volunteers Host Country Distributing Country (Perception of distributing country and agency) Has understanding of KOICA activities increased through volunteer work? Has the sustainability of KOICA work increased through volunteer work? Has the perceived image of Korea improved through volunteer work? (Nurturing global talent) Has global talent (possessing foreign language skills, receptivity and adaptability to foreign cultures) that can contribute to the good of humanity been nurtured? Are you continuing work using your ʻglobal talentsʼ? (Contributing to the improvement of living standards) Have you contributed to poverty reduction and sustainable development of the host country? Is the impact of volunteer work on cooperating agencies and local community sustainable? (Improved relationship with host country) Has an amiable relationship been established through volunteer work? Has trust been established between the host and distributing countries? Methodology and Source Method: Deduction based on the results evaluation of Outcomes Source: Interview results, Questionnaire results 18 Comprehensive Evaluation of KOICA WFK Program

27 4. Evaluation Methodology The evaluation team conducted the evaluation using literature analyses, interviews with relevant personnel and questionnaires. Literature analyses focused on analyzing the evaluation approach of international volunteer projects of other international aid agencies and the overall situational analysis of the KOICA WFK. The interviewees were KOICA WFK personnel and relevant parties. The aim of the interviews were to gather opinions about project strategy, implementation and implementation systems and collect information that was hard to access through existing literature. Interviews in the distributing and host countries consisted of focus group interviews and one-on-one in-depth interviews. The interviewees in Korea were personnel at the WFK Headquarters, WFK Training Center and employees in charge of managing the volunteers. External parties relevant to the project such as, the volunteers, KOVA (Korea Overseas Volunteers Association), KCOC (Korea NGO Council for Overseas Development Cooperation) were also interviewed. Focus group interviews were conducted in host countries targeting the local cooperating agency employees (local and national government, local residents), local KOICA office employees and volunteers that were active at that time The survey was conducted Nov over a period of two weeks. The overall implementation system's relevance and efficiency were evaluated and the surveys targeted volunteers who returned to the distributing country after working in Rwanda and Cambodia January 2012? present. The surveys also targeted the heads of cooperating agencies and co-workers who collaborated with the volunteers over the same time period. A Computer-Aided Web Interview was conducted online to interview and survey these parties. Evaluation Overview 19

28 [Table 1-6] Summary of Evaluation Method Stage Evaluation Method Output Stage 1. Creating an evaluation framework Creating an evaluation framework for situational analysis and results evaluation Method: Literature Analysis Target: M&E information on international volunteer projects from other aid agencies Evaluation framework Analysis of KOICA WFK strategy Method: Literature Analysis Target: KOICA WFK's project implementation strategy and respective country strategies Stage 2. Situational analysis Analysis of KOICA WFK's implementation situation Method: Literature Analysis, Interviews (host and distributing countries), Surveys Target: Literature on KOICA WFK's implementation system /relevant personnel from WFK HQ, relevant KOICA local office personnel, volunteers who have returned to Korea. Results from situational analysis according to evaluation category Stage 3. Results Evaluation Results analysis for volunteers Results analysis for host country Results analysis for distributing country Recommendations for improving the results the KOICA WFK Method: Interviews (host and distributing countries), Surveys Target: Volunteers who have returned to Korea, local cooperating agency personnel Results from situational analysis according evaluation category Stage 4. Recommendations Recommendations for improving the results the KOICA WFK Recommendations for managing the results the KOICA WFK Method: Make recommendations based on analysis results for Stage 1 and 2. Target: Analysis of results from Stage 1 and 2. / M&E information on international volunteer projects from other aid agencies Recommendati ons for improving KOICA WFK results and results management 20 Comprehensive Evaluation of KOICA WFK Program

29 5. Limitations of the Evaluation Firstly, due to a lack of sufficient evidence and data there were limitations in evaluating the project's impact according to the established theory of change and evaluation framework. However, the evaluation was complemented by the data gathering carried out by the host country's own evaluation and the indicator records from the volunteers who are currently active in the field. Secondly, there were limitations in conducting an objective evaluation as the personal information of the returned volunteers is protected. Therefore, to make the evaluation more objective, efforts were made to gather objective opinions and other methods of interviewing were sought. Thirdly, differences in individual opinions were not accounted for. Most of the evaluation results were derived from interviews with volunteers and local cooperating agency personnel. To overcome this limitation, the evaluation results from the interviews and existing data were also included. Evaluation Overview 21

30

31 Ⅱ. Theoretical Background of the Evaluation 1. Trends in Evaluating International Volunteer Program 2. Evaluation Methodology for International Volunteer Program

32

33 Chapter Ⅱ Theoretical Background of the Evaluation 1. Trends in Evaluating International Volunteer Program The impact and effectiveness of international development volunteers has been called into question by the UNDP and the German government. From the mid-2000s, the number of international volunteers has been on the rise and there have been increasing efforts to evaluate the effectiveness of international volunteer projects. Despite this, most evaluations have been conducted on a regional or project basis, and most implementing agencies have also stopped issuing guidelines that quantify and systemize evaluations and give examples of them. The reason for this is that it is difficult to establish an integrated evaluation model or standard, because such projects differ vastly in their goals, time frame, size and character. Against this backdrop, the ILO (International Labor Organization) and John Hopkins University's Center for Civil Society Studies have cooperated to create a manual that states the definition of volunteer work, and the goals, standards and approaches for evaluating it. To evaluate data, the ILO states that the Opportunity-Cost Approach and Replacement-Cost Approach are appropriate to utilize. Both of these approaches aim to measure the economic value of volunteer work. On the other hand, there are also attempts to emphasize that the value of volunteer is not merely economic, but lies in its importance to society in general therefore there is a need to create an evaluation framework that measures Theoretical Background of the Evaluation 25

34 these changes. One of the leaders of this effort is the UNV (United Nations Volunteers). In 2010, the UNV recognized international volunteer organizations as a type of expanded civil society, and facilitated their participation in international development. In doing so, they emphasized their importance. International volunteer organizations are now expected to contribute positively to the SDGs - especially SDG 16 (Peace, Justice and Strong Institutions) and SDF 17 (Global Partnerships). In affirming the social value of volunteer work, instead of simply its economic value, there have also been attempts to combine the theoretical underpinnings of evaluation and social capital, in order create a new perspective in measuring social progress. International organizations such as the UNV, World Bank and international aid agencies are specifying this social concept and creating evaluation tools to measure it. The United Kingdom developed a Volunteering Impact Assessment Toolkit in 2004 that can measure the various types of capital that can have social and economic influence. Through pilot evaluations it assessed the validity of such a tool (Jarvis & Wilson, 2007). This tool aims to evaluate the impact or effect of volunteer work on human, economic, social and cultural capital. The evaluation model is significant in that it elaborates on the economic and cultural value of volunteer work - however it is important to give it careful consideration before applying it, as it does not incorporate a development perspective. Separate from the question of what to emphasize during an evaluation, is the matter of which of the results can be evaluated. Sherraden, Lough, & McBride (2008) identified factors that could influence volunteers, the character of the volunteer program, the capacity of the individuals and institutions and the character of international volunteer programs. They also categorized the results of international volunteer work into recipient country, volunteers and donor country. 26 Comprehensive Evaluation of KOICA WFK Program

35 [Table 2-1] Results of International Volunteer Project: Recipient country, Volunteers, Donor country Recipient country Volunteers Donor Country 1 Social, economic, environmental, political situation 2 Intercultural knowledge and skills 3 Knowledge and understanding of international affairs 4 Participation of the international community 5 Reciprocity and Dependence, 6 Capacity of the recipient agency 1 Work experience and skills 2 Personal development 3 Adaptability to different cultures and language skills 4 Understanding and comprehension of international affairs 5 Participation of civil society and the international community 1 Human capital 2 Adaptability to different cultures 3 Intercultural knowledge and skills, 4 Social, economic, environmental, political situation 5 Participation of the international community 6 Capacity building for the recipient country Source: Sherraden, Lough & McBride Effects of International Volunteering and Service : Individual and Institutional Predictors. Lough, McBride, and Sherraden (2009) gave specifications on the evaluation framework they developed, by stating that surveys should be conducted in order to evaluate the interrelatedness and importance of various elements. To do an analysis of the elements and verify the validity of the indicators in the future, there is a need for personal information regarding the volunteers (ex) foreign language skills, gender, cultural diversity, international identity, participation in local community, reason for applying, and whether they are individual donors or not. The authors emphasize that the analysis of results and identification of elements that directly and indirectly influence volunteer work, acts as important information in carrying out volunteer activities and evidence-based policies for these programs. Theoretical Background of the Evaluation 27

36 2. Evaluation Methodology for International Volunteer Program The evaluation method for each agency s volunteer project varies according to the project's goals and the evaluation's goals. It is useful to look at examples of the main volunteer projects of other donor countries such as: Peace Crops (USA), JOVCV (Japan), ICS (UK), AVID (Australia), as well as the various discourses, approaches and actual evaluation methods that they implement. Doing this made it possible to deduce recommendations that would help develop a framework that could evaluate the WFK properly. Peace Corps is using a new method of data collection in order to determine whether their project goals have been achieved, as well as the usual in-depth interviews, local investigation and literature analysis. Some examples of these new methods are the Global Counterpart Survey, which surveys the main project personnel, and the Host Country Survey of local employees hired by the Peace Corps. There is also the Volunteer Reporting Tool, where improvements have been made to the input system used by volunteers when they enter their data directly. JICA usually evaluates their JOCV international volunteers by conducting a survey every three years, but some of the surveys (for example, surveys for recipient agencies and active volunteers) are conducted every quarter. An interesting fact is that there are surveys for recipients and the general Japanese public, as well as surveys for volunteers 2 years and 5-7 years after they have returned. On the other hand, JICA's evaluation is dependent on surveys and swayed towards quantitative evaluation. The objectives of the UK's ICS Programme Evaluation are 1) Usage for demonstration projects and evaluating outputs 2) Making recommendations to 28 Comprehensive Evaluation of KOICA WFK Program

37 improve the design and implementation of the ICS Programme 3) The value for money for every stage of the programme. The methods used are satisfaction level surveys for returned volunteers and relevant parties, interviews (via phone, , face-to-face), observation, analysis of budget reports and systems and on-ground research in six countries (Bolivia, El Salvador, India, Nepal, Kenya, Tanzania, Uganda). They serve to evaluate the primary objective of the ICS Programme, which is the active citizenship of the participating volunteers. Another characteristic of the evaluation is that it also considers contributing to the regional development as one of its main objectives. Evaluation of Australia's Volunteers for International Development (AVID) focuses on evaluating factors such as strategic alignment with the Australian aid programme, existence of programme support and support plans, management of project implementation and the extent of contribution to development by volunteers. Evaluation methods include: literature analysis, surveys for 1,361 returned volunteers and recipient agencies, schematization of volunteer placement, media analysis and on-ground research in Cambodia, Vietnam and Solomon Islands. In the context of a changing international development landscape the role of international volunteers is becoming increasingly important. However, serious discussion on the need to evaluate the impact and results of international volunteer work has only begun recently. At this stage there are no international standards on volunteer program evaluation that are official or reliable, such as the OECD DAC etc. Given this situation, international organizations such as the ILO, UNV and World Bank are creating their own evaluation frameworks based on their respective aims and perspectives, such as: economic value, social cohesion value, social capital formation value and etc. On top of this, aid agencies such as the Peace Corps, JOCV, VSO and AVID have devised their own frameworks based on their current programs, organizational capacities, evaluation Theoretical Background of the Evaluation 29

38 capacities, and evaluation goals. By assessing the various evaluation methods that are being used and the discussions and methods surrounding them, the evaluation team could deduce recommendations that helped in development of a framework, one that could properly evaluate of the characteristics and results of World Friends Korea. Firstly, the methods used by international aid agencies to evaluate their international volunteer projects are shown in the table below. Their methods are evolving from evaluating the individual volunteers to evaluating whether the recipient country's demands are being met, and the level of contribution to the capacity building of the recipient and country and agency. The range of targets in the evaluation should be expanded from the volunteers to include the personnel from the recipient country/agency and local residents. It is necessary to move away from a volunteer-centered evaluation framework in order to use a framework that targets both the donor and recipient country and evaluates the results from multiple angles. If such a framework is developed, then it will be able to explain the connection between results and implementation, and incorporate the perspectives of various actors within the project, and break away from one-sided evaluations. [Table 2-2] Main Evaluation Methods of International Aid Agencies Agency United States - Peace Corps Evaluation Method Goal: Contribute to peace and human development Direction: Traditional evaluation based on outputs Emphasis of impact on recipient country Evaluation Responsible: Office of Strategic Information Research was established to evaluate the impact of volunteers from the recipient country's perspective Method: Evaluate results by using quantitative methods to measure numerical figures (ex) no. of volunteers active) Focus: Use multi-faceted methods to gather information from local residents and personnel from the government and relevant organizations. 30 Comprehensive Evaluation of KOICA WFK Program

39 Agency Japan - JOCV United Kingdom - VSO Evaluation Method Goal: Achieving the partner country's development goals Direction: Mid-to-long term impact evaluation targeting volunteers 3 years after they have returned. Evaluation Responsible: Commission external experts to conduct the evaluation Method: Surveys and interviews for active volunteers and offices abroad Focus: Conducting a mid-to-long term impact evaluation after the volunteers have returned (3 years after / 5-7 years after) Goal: World without poverty Direction: Collecting interviews and cases about project implementation and management systems Method: Data and information from the actual implementation of the volunteer projects, interviews with volunteers who have returned Focus: Focus is on communication, relationships during the project implementation and collecting quantitative research and success stories, rather than efficiency and effectiveness of the project. Source: JICA. (n.d). Japan Overseas cooperation volunteers; VSO Evaluation of the International Citizen Service. Secondly, the evaluations are solely conducted in quantitative or qualitative methods or a mixed quant/qualitative method. Instead of using a dual-evaluation method that assigns quantitative methods to measure economic value and qualitative methods to measure social value, there needs to be an approach that reflects both aspects. International volunteer projects require a lot of expenses, so the cost-benefit ratio should not just involve quantifiable value but outputs such as whether the local residents and agencies have been given opportunities to develop their capacity, and if they helped their own people grow or not. Lastly, the personal characteristics of the parties involved in the projects (such as the volunteers) need to be factored in, considering the diversity of volunteer projects. Factors such as gender, age and reason for applying are elements that could affect the project results. If these elements are not considered, they will not play a role in the overall policies or improvements that need to be made in the project stages (ranging from recruitment to training and implementation). In Theoretical Background of the Evaluation 31

40 past, most evaluation focused on surveys targeting the volunteers. This evaluation aims to go deeper and analyze the various personal characteristics of the volunteers, in order to make more concrete and specific recommendations. 32 Comprehensive Evaluation of KOICA WFK Program

41 Ⅲ. Implementation Situation of KOICA WFK 1. Strategy 2. Implementation Situation

42

43 Chapter Ⅲ Implementation Situation of KOICA WFK 1. Strategy The WFK's vision is "A better world through sharing and learning". Its goals are: 1) To improve the quality of the life of residents in developing countries. 2) To increase cooperation and mutual understanding between developing countries and Korea. 3) To achieve self-realization and growth through service activities. This part looks at whether an appropriate strategy was established during the planning and implementation process. To do this, we analyzed the alignment between WFK Strategy and Korea's ODA Strategy, the alignment between WFK Strategy and the hosting country's demand, the alignment between WFK Strategy and international aim norms, and whether the WFK strategy has actually been implemented. [Table 3-1] WFK Vision ODA Mission WFK Vision WFK Mission WFK Main Values Reduce poverty in low-income countries and improve the standards of living. Realize sustainable social and economic development and solve global development issues through cooperation with the international community. Global prosperity LEAD with WFK LEADers A better world through sharing and learning Learning Experience Accountability Diversity WFK Goals Improve the quality of the life of residents in developing countries Increase cooperation and mutual understanding between developing countries and Korea Achieve self-realization and growth through service activities Implementation Situation of KOICA WFK 35

44 WFK Strategic Goals Roles of WFK Volunteers Strengthen the capacity of cooperating agencies and meet their demand Shares knowledge and experience Increase in understanding between countries Facilitates change in local community Enhanced global capacity of volunteers High quality activity support for better project results Intermediary for cultural exchange Raise brand awareness of WFK Challenges oneself to a new life Source: WFK Comprehensive Planning TF World Friends Korea Comprehensive Roadmap for International Volunteers (Draft). There may be conflicting perspectives between the host country, distributing country and international cooperation concerning the direction of the three KOICA WFK goals. The first goal, which is "To improve the quality of life of residents in developing countries", aligns well with Section 3 Clause 2 of Korea's International Development Law which stipulates "... reduction of poverty and improvement of standards of living in low-income countries". The second goal, "To increase cooperation and mutual understanding between developing countries and Korea" does not differ greatly from general international development projects. The third goal, "To achieve self-realization and growth through service activities" does align with the Korean government's plans in 2008 to nurture young global leaders. However, it can be said that this is a goal that is centered solely on the distributing country, and does not align with international trends in aid effectiveness. Concerning the KOICA WFK Strategy shared amongst relevant project personnel, it seems that a consensus on the overall project goals formed once the agencies begun to share their strategies. In 2013, a 5-year roadmap ( ) that included the WFK comprehensive project long-term vision, goals and expected impact was created and contributed to the development of this consensus. 36 Comprehensive Evaluation of KOICA WFK Program

45 The KCOC, which is contracted by KOICA to carry out NGO volunteer work, has the goals of 1) Contributing to the host country and the local community's socioeconomic development and welfare 2) Mutual understanding and cooperation between the host country and local community 3) Improving aid effectiveness in civil society and capacity building for international development NGOs 4) Personal growth for individual volunteers and human resources development. These are largely in the same context as KOICA WFK's comprehensive project goals. There is a system for sharing documents on the strategic direction of each project goal, region and area among WFK HQ personnel, implementing agencies, project participants (volunteers, managers). Other systems for sharing include: shared training curriculums, cooperating agency meetings, department meetings, local evaluation meetings, local training and manager meetings. However, some of the volunteers are not aware of the project strategy at all - they have different levels of inter-sharing or understanding. Sufficient comprehension of the strategy among volunteers seems necessary. KOICA states the WFK strategy and vision on its website and also during its training. During the interview with relevant personnel, when asked whether the three goals of the WFK were achievable, some said that all the goals were achievable, but there were also some who said that it would take more time and effort depending on the goal. In terms of the institutional, policy foundations for implementing the project, they are appropriate and systematic in Korea and throughout WFK, and are functioning as a whole system instead of disparate elements. The KOICA WFK's goal of providing development cooperation training and practical experience to 30,000 persons from 2013 to 2018, aligns with the current Korean government's National Policy Priority No. 133, which is to nurture young global leaders. However, the fact that the volunteer projects are being directly Implementation Situation of KOICA WFK 37

46 linked with another national priorities, such as job creation, could weaken the true purpose of volunteer work and act as a roadblock to project implementation. In late 2011, a WFK Guideline for Hosting Countries was created for the first time, and it selected a total of 49 countries - 15 Priority Dispatch Countries, 20 General Dispatch Countries and 14 Strategic Dispatch Countries. Under this strategy, 60% or more of all countries are Priority Dispatch Countries, 30-40% are General Dispatch Countries and 10% or less are Strategic Dispatch Countries. In 2013, the ratio of Strategic Priority Countries was 80.2% (24 countries). The current 45 countries were selected in December 2013, and KOICA has a separate country strategy for 24 of them. (1st round: / 2nd round ). In the Relevance section of the dispatch strategy, one can find the country strategies. As shown in the box below, the country strategies are being established through policy discussions with cooperating countries and by considering their development strategies, and also KOICA's own strategy and focus areas. However, the strategies are long-term country strategies that last 3 years, so the strategic planning process needs to improved and re-established. [Box] KOICA WFK Regional and Country Strategies Regional Strategies Asia: Considering the geopolitical location and political, diplomatic relationships with Korea, Asia is a primary focus region. 55% of volunteers are dispatched to Asia. Also, in efforts to increase support for ASEAN countries, it will focus on capacity building for human resources as a part of its strategy to stimulate growth. Africa: The highest demand for development cooperation comes from Africa, therefore 23% of volunteers are dispatched to the region. Support is focused on low-income countries and the main goal was to contribute to the achievement of MDGs (prior to 2015). Latin America: There is a suitable environment for volunteer activities and living conditions in Latin America, and therefore 15% volunteers are dispatched to the 38 Comprehensive Evaluation of KOICA WFK Program

47 region. The strategy in Latin America is to strengthen activities in decreasing the poverty gap and the disparity between the rural and urban areas. Middle East and CIA: 7% of volunteers are dispatched to the region. Major countries that have religious and social customs for volunteer work are supported, and the focus is on science technology and establishing industry infrastructure. Country Strategies Establishing the strategy: The country strategies for KOICA WFK were established with the goals of 1) Strategic implementation tailored to each country 2) Encouraging cooperating governments and volunteer agencies to be more proactive in project participation 3) Developing and sharing programs that accumulate and disseminate project results. For the establishment of country strategies, KOICA offices around the world have conducted workshops with volunteer project personnel from cooperating agencies, among other regular policy cooperation. They analyze in detail the development strategy of cooperating governments, achievement of the MDGs (and now the SDGs), levels of poverty and establish strategies based on this analysis. Criteria for selecting countries: Among the countries that have been selected as non-conditional aid distribution countries, the ones that have the most appropriate conditions for activities and living conditions are chosen. Then it is narrowed down to countries that have 10 new volunteers or more every year. In addition to this, support is focused on OECD DAC's Low-to-Medium Income Countries. For countries that had strategies before, the project results are monitored and the number of volunteers is chosen based on this. In the case of new countries, volunteer activities and living conditions are considered while increasing the number of volunteers gradually over time. Implementation evaluation: Implementation evaluations for country strategies are divided into yearly evaluations and comprehensive evaluations. For yearly evaluations, the KOICA offices abroad hold yearly workshops with local cooperating agencies and local evaluation conferences with volunteers, in order to monitor results per area. The objective indictor for ʻʻImplementation and expected impactʼʼ for each designated area is then established. HQ conducts surveys targeting cooperating agencies and volunteers, to monitor the main results from country strategies. The comprehensive strategy involves synthesizing 4 yearsʼ worth of yearly evaluations into strategic implementation results and reporting to HQ. HQ refers to the comprehensive report and reflects it on their future strategies and lessons. Implementation Situation of KOICA WFK 39

48 [Table] 2014 WFK Volunteer Project Countries Category Countries Total 45 Countries Asia 8 Nepal, Laos, Mongolia, Bangladesh, Vietnam, Indonesia, Cambodia, Philippines Priority Rwanda, Morocco, Senegal, Ethiopia, Uganda, Africa 7 Countries Cameroon, Tanzania (15) Latin America 1 Paraguay CIS 1 Uzbekistan Asia 6 East Timor, Myanmar, Solomon Islands, Sri Lanka, Thailand, Philippines General Africa 4 Ghana, Mozambique, Egypt, Tunisia Countries Latin Guatemala, Dominican Republic, Bolivia, Ecuador, 8 (20) America El Salvador, Honduras, Colombia, Peru Middle East 1 Jordan CIS 1 Kyrgyzstan DR Congo, Nigeria, Kenya, Democratic Republic Africa 3 Strategic of Congo Countries Central (14) America 2 Costa Rica, Panama CIS 3 Azerbaijan, Ukraine, Moldova Source: WFK Comprehensive Planning TF WFK Comprehensive Project Plan [Table] Priority Countries / Countries that have WFK Country Strategies (2014) Region Asia (11 Countries /10 Countries) Africa (8 Countries /7 Countries) Latin America (4 Countries /Countries) CIS (2 Countries /1 Country) Oceania (1 Country/-) ODA Priority Countries (26) Vietnam, Indonesia, Cambodia, Philippines, Bangladesh, Mongolia, Laos, Sri Lanka, Nepal, Pakistan, East Timor Ghana, DR Congo, Nigeria, Ethiopia, Mozambique, Cameroon, Rwanda, Uganda Colombia, Peru, Bolivia, Paraguay Uzbekistan, Azerbaijan Countries with WFK Country Strategies(24) Priority Countries are in bold. Vietnam, Indonesia, Cambodia, Philippines, Bangladesh, Mongolia, Laos, Sri Lanka, Nepal, Thailand Ethiopia, Cameroon, Rwanda, Morocco, Senegal, Egypt, Tanzania Peru, Bolivia, Paraguay, Dominican Republic, Ecuador, El Salvador Uzbekistan Solomon Islands - Source: WFK Comprehensive Planning TF WFK Comprehensive Project Plan 40 Comprehensive Evaluation of KOICA WFK Program

49 The WFK Comprehensive Project Plan maintains the support focused in the Asian region, but also gives stipulates a more extensive dispatch strategy that targets low-income countries in Africa and etc. to achieve the MDGs. However, when the plans from 2012 and 2013 are compared, in 2012 the support for low-income countries was 40.2% (20 countries) but reduced to 38.5% (19 countries) in The number of mid-to-high income countries decreased from 9.8% (13 countries) in 2012 to 7.7% (12 countries). The alignment between the KOICA WFK Strategy and the recipient country's demand emphasizes volunteer work and focused support based on prioritized development needs. In the 2015 the WFK Volunteer Project Implementation Strategy, education, public health and public administration are stated as focus areas, which also aligns with KOICA s focus on education, public health, public administration, agriculture, industrial energy, cross-sector and climate change. From 1990 to 2013, among the 11,007 volunteers that were dispatched, the ratios dispatched in to focus areas were the following: education (54.3%), public health (13.9%) and agriculture (12,8%). In the year 2013, the numbers were: education (55.5%), agriculture (15.3%), public health (12.2%) and public administration (11.5%). The sample countries Rwanda and Cambodia were found to have alignment between strategy and demand. The strategies were based on both governments national development strategies, requests based on demand and Korea's country partnership strategy. KOICA WFK's country strategies make predictions for demand, but differ from one another on how specific they are. In terms of alignment with international aid norms, the WFK goals and strategies were in large part referring to the main currents in international development. They were also trying to refer to the UNV and other international volunteer organizations. Furthermore, according to the 2014 WFK Comprehensive Project Plan, they were trying to bolster their expertise by using Korean and Implementation Situation of KOICA WFK 41

50 international networks to exchange information and cooperate with other volunteer agencies. For example, in 2013, KOICA signed an MOU with Peace Corps and joined the global network for international volunteers 'FORUM' as a full member and then participated in the UNV Technical Workshop. In 2015, KOICA was working with a greater number of experienced international volunteer agencies to create evaluation indicators for common goals such as the SDGs and other global issues. 2. Implementation Situation (KOICA WFK Implementation System) To evaluate the WFK's relevance and efficiency, a strategy and processes evaluation matrix that summarizes the implementation method and characteristics was created to use for analysis. The implementation system is visualized in the following figure (Figure 3-1) - consisting of one project head and three sub-teams in the WFK team and the WFK Training Center. [Figure 3-1] KOICA WFK Implementation System (Organization Chart): Apr 2015 Source: KOICA WFK-KOICA International Volunteer Project Implementation Plan 42 Comprehensive Evaluation of KOICA WFK Program

51 The simple quantitative difference between the amount of work KOICA WFK HQ has and the number of employees it retains is enough of a reason to presume limits to its work. HQ administers a budget of approximately 10 billion Won and recruits, hires, trains and manages 1,400 volunteers. 37 employees in HQ (as of April 2015, including full-time employees and contractors) are not enough to manage all the work they have in an efficient and effective manner. At the KOICA Rwanda office, four managers are supporting 45 volunteers (approximately 12 volunteers per volunteer). On the other hand, at Peace Corps, there are approximately 30 managers for every volunteers (3 volunteers per manager). Local offices are also responsible for managing volunteers, along with HQ, therefore in order to achieve qualitative growth along with quantitative growth, now is an important time to hire more management personnel. The fact that during the process of restructuring the international volunteer projects, Korea's main aid agency KOICA and the WFK were assigned to jointly manage the Korean government's international projects could be beneficial for aid effectiveness and improving project efficiency. However, under the current implementation system, only the budget and brand have been unified and the actual project implementation is conducted by 7 different agencies. This inevitably creates structural limitations to ODA projects and administrative inefficiency. Cooperation amongst the relevant personnel within KOICA WFK HQ seems to be seamless in general. Usually there are weekly and monthly meetings, face-to-face agreements and document sharing. The frequency of meeting is usually 1-2 times a week. The team manager(s), team head(s) and head of project meet once a week, and the section heads meet once a week as well. For unofficial matters, personnel use conventional cooperation methods depending on the issue. HQ and the implementing agency (the Comprehensive Planning TF team) meet four times a year, as well as there being many instances of informal cooperation with the implementing agency as well. Implementation Situation of KOICA WFK 43

52 To strengthen the expertise and partnerships of KOICA WFKs, KOICA is expanding information-sharing and mutual cooperation projects with international organizations and volunteer agencies. In Korea, KOICA is cooperating with private companies to connect their corporate capacity with volunteer activities. However, in the country strategies, there is no specific plan for the local offices to establish a network for strengthening project expertise. Through interviews, it was found that problems amongst volunteers occasionally arise, because most of the volunteers are young adults in their 20s and 30s with little training, they lack the capacity to function efficiently in their assigned countries. Therefore, it is important to establish a network where volunteers can be re-trained in their professional field. [Figure 3-2] KOICA WFK Implementation Steps Source: WFK Comprehensive Planning TF WFK Comprehensive Project Plan (KOICA WFK Implementation System) KOICA's WFK Implementation System is made up of 5 stages: Demand Assessment, Recruitment and Selection, Training in Korea and Hosting Country, Activity Support, Post-Project Management. a) Demand Assessment Normally, the demand assessment carried out on KOICA WFKs is done by KOICA offices abroad. The assessment form asks for the following: summary of dispatching agency, dispatch background, job description, required certifications for volunteers and volunteer activities. Job description is one area that requires 44 Comprehensive Evaluation of KOICA WFK Program

53 a detailed breakdown of the expertise of volunteers, even within a single job group. This is because it is an area that can improve the alignment between the host country's demand and volunteer skills. However, according to <Interview 1>, the local agency personnel and co-workers have a hard time understanding the categories, so there are instances where the matching isn't done properly. A lack of understanding of local demand from the local agency's side leads to misalignment between the host country's requests and the skills of the volunteers. <Interview 1> "The local community doesn't know what kind of skills and background the volunteers should have. For instance, a vocation training institution asked for a volunteer who could do computer design. But the institution employees don't know anything about computer design. They think that since computer design is trendy, it would be good to ask for a volunteer who can do it. This is how the requesting agency fills out their request forms". (K, a returned volunteer) According to the WFK Comprehensive Satisfaction Survey, the host country's demand and the volunteers were relatively misaligned compared to other types of volunteer work. Compared to mid-to-long term advisors who need to possess specialized expertise, KOICA WFK volunteers only require certain certifications or higher education studies in a specific field. There could be instances where the KOICA WFK volunteers are inappropriate for the volunteer work assigned - the primary cause for this is lack of communication between the local agencies and volunteers themselves on the specific jobs they are expected to carry out. In the same survey asking about causes for dissatisfaction from the local agencies and residents, 40% said that volunteers were not assigned to the appropriate agency and 24.4% said that they themselves (the volunteers) lacked expertise. Other answers included lack of time and facilities and inadequate demand assessment. On the matter of inappropriate assignment to agencies, according to <Interview 2>, a lack of expert personnel that volunteers can interact with, along with a lack Implementation Situation of KOICA WFK 45

54 of activities, creates physical and psychological difficulties for the volunteers from the beginning of their duty. <Interview 2> I went as a culinary volunteer, but nobody there had any expertise in cooking. So they had no criteria for judging whether the place I was dispatched to had the adequate environment for me to do professional work. They had just general knowledge and selected me based on that, but just like what the other volunteers experienced, the environment there was extremely lacking..." (Y, returned volunteer). Currently, WFK HQ is developing a new method of demand assessment in cooperation with expert professors from vocational training institutions (Korea University of Technology and Education, Sinhan University, Cheonan Yeonam University), based on National Competency Standards (NCS), which will improve the relevance of volunteer work. The new assessment will evaluate the conditions and facilities in the working environment, as well as the education levels there. To improve the effectiveness of demand assessment, the demand assessment of cooperating agencies needs to be institutionalized and a stronger strategy is needed when selecting cooperating agencies. Currently, demand assessment for volunteers is conducted through the managers (of the volunteers) visiting the respective regions and agencies they are responsible for. However, there are no clear criteria for demand assessment and the assessment relies solely on the abilities of the managers, therefore it is necessary to establish clear and specific criteria. There is a high demand for technical education, but there has been some difficulty in dispatching volunteers with such backgrounds. In a scoring evaluation conducted with the local agency personnel, the demand assessment by hosting countries scored 4.28 out of 5, in between "Yes" and "Very much so". Specifically, 20.7% answered "Yes" and 62.1% answered "Very 46 Comprehensive Evaluation of KOICA WFK Program

55 much so", adding up to 82.8% who had a positive response. Those who answered below "Yes" were 17.2%. When the respondents were categorized into personnel who had experience in cooperating with volunteers, agency type and type of work, the survey results showed that the perception of demand assessment differed according to what kind of work they were responsible for (as shown in the table below). When asked about alignment between the demand assessment and the volunteers, the result was 4.06 out of 5, close to "Yes". 44.8% answered "Yes" and 33.3% answered "Very much so", adding up to 78.1%. 21.9% answered below "Yes". Most of the volunteers believed that the demand from cooperating agencies was aligned with their own volunteer work. When the survey volunteers were asked whether their major, aptitude and preferences were considered in where they were dispatched, the score was 3.96 out of 5, with 38.5% answering "Yes" and 37.5% answering "Very much so", adding up to 76%. 24% answered below "Yes". Most of the volunteers believed that their major, aptitude and preferences were considered in where they were dispatched to. When the survey asked what was important in fulfilling demand by cooperating agencies and local societies, 51% said "Systematic demand assessment targeting cooperating agencies and local societies". 34% said "Selecting volunteers with expertise", followed by "Consideration for the major, aptitude and preferences of the volunteers". Other responses included "The expertise of the volunteers is important, but the mindset of the volunteers is even more important". Implementation Situation of KOICA WFK 47

56 [Figure 3-3] Important factor for fulfilling demand by cooperating agencies b. Recruitment and Selection During recruitment and selection, various outlets and methods are using according to the WFK Promotion Guidelines. The recruiting event has been expanded to attract candidates with expertise and a passion for volunteering. Most of the volunteers that were surveyed felt that the recruitment and hiring procedures were implemented adequately. [Table 3-2] Adequacy of Recruitment and Hiring Procedures Category N Average Standard Deviation Adequacy of Recruitment and Hiring Procedures [Figure 3-4] Adequacy of Recruitment and Hiring Procedures 48 Comprehensive Evaluation of KOICA WFK Program

57 According to <interview 3>, volunteers pointed out that there were limitations in reviewing their quality through personality and aptitude tests. As shown in table above, the average score for the survey was 3.45, meaning most volunteers thought that the recruitment and hiring process was "Average". Out of the various implementation systems, this is a relatively low score. Evaluation of the candidate's expertise is relatively objective, by reviewing the applications and conducting interviews. However, considering the large number of volunteers that are dispatched every year, there are some questions being raised about the character of the volunteers (in regard to respect for local residents and etc.). <Interview 3> highlights that there are limits in evaluating a candidate's character and aptitude during the selection process. <Interview 3> "Even now, KOICA conducts character and aptitude evaluations, but they are not being done thoroughly. During volunteer training in Korea, there are some who are filtered and asked to leave, but I think it's possible to hide things for a month. It becomes a problem when unqualified people get dispatched. So I think it's right to filter people. But sometimes unqualified people do get dispatched..." (J, returned volunteer) Requirements for systemization of volunteer recruitment and selection methods included: "Improving the selection system through training" (47%), "Stricter selection system for volunteers" (28%), "Stricter standards for selecting volunteers" (25%) Implementation Situation of KOICA WFK 49

58 [Figure 3-5] Requirements for systemization of volunteer recruitment and selection c. Training in Korea and Hosting Country The amount and period of volunteer training has decreased in recent years. Training facilities are packed beyond capacity, and there is a limit to the space that can be used for training. Therefore, increasing the quality of the training is an issue that needs to be discussed. Furthermore, despite general satisfaction in the training contents, it was found that more resources are needed to practice the local language, during and after both types of training. Upon on-site inspection, it was also found that because the training for Cambodia and Rwanda was compressed in to a short time period with limited content, volunteers felt pressured. There were also calls for training that could actually be applied on the ground. In <Interview 4> there were concerns about generational differences with the senior volunteers, overcoming cultural differences and the need for specialized training for capacity building. <Interview 4> "Only volunteers with 10 years in the field can be considered senior volunteers, but sometimes there are general volunteers who are older than the average age. These volunteers usually held high positions in Korean organizations and were treated as such, so they come to the host country and ask to be treated like they were in Korea. They 50 Comprehensive Evaluation of KOICA WFK Program

59 ask for things from their fellow teammates or other organizations. They don't talk that way at first and they act modest. But when it comes to real problems, they say things like 'Why are you treating me like this?' and so on... I think it's the elder volunteers that need more training..." (C, volunteer) In the survey asking volunteers about their perceptions of the training, the training in Korea scored 3.38 in relevance and 3.48 in adequacy of period and amount of training. For training in host countries, relevance was 3.74 and adequacy of period and amount of training was Generally, most volunteers scored training above "Average", but the perception of the training in the host countries scored lower than the Korean counterpart. When asked what improvements should be made in the training, 43% answered "Local language skills", followed by "Increased training on development cooperation" (21%) and "Training on hosting country"(18%). There were also other responses, such as "Safety training" (3%). [Figure 3-6] Improvements required in for training Implementation Situation of KOICA WFK 51

60 d. Activity Support The volunteer management system used by KOICA offices abroad and volunteer managers is supported systematically compared to other volunteer projects. However, each manager has to manage 24 volunteers, which is a much higher number than Japan's JOCV (17 volunteers per manager) and Peace Corps (11 volunteers per manager). Therefore, managing the volunteers has its limits. For safety management, group training on safety is being expanded, along with support for comprehensive safety inspections and the provision of safety equipment and security systems for volunteers. Specifically, the safety management system for volunteers included safety management guidelines for each country and more detailed safety management procedures within in. The latter includes the safety situation of each country, information on natural disasters, evacuation plans, measures to be taken in each emergency situation and safety case examples. There is also group safety training, as well as these guidelines and manuals. A safety coordinator (ex-police of security expert) also examines the overall safety situation for volunteers. Volunteers are given safety equipment on a regular basis and security systems for their accommodation (reinforced windows, specialized safety locks). These are all efforts to improve safety management for the dispatched volunteers. In terms of administrative support, $2,500 worth of volunteer activity supplies were distributed per volunteer, and the satisfaction level for this support went up from 77 (points) in 2012 to 87 (points) in This was due to improvement in management and the implementation of service delivery by the subcontracted procurement companies, as well as consistent project cooperation. Approximately 200 projects suggested by the volunteers were implemented and improvements were made to the local project screening committee (formation and working period of the committee) and the evaluation forms (project proposals, result 52 Comprehensive Evaluation of KOICA WFK Program

61 reports, evaluation indicators). The scope of the committee was also expanded from $30,000 or over, to compulsory projects that cost $10,000 and over. Workshops for improving capacity building for project management are being conducted, as well as monitoring for local projects, monthly project progress reports and management of low progress projects. [Box] Activity support systems for KOICA and other agencies KOICA JICA Peace Corps Financial Support While Active Living cost for Rwanda is $580. Accommodation cost for the urban region is $560 and $400 for rural areas. Apart from support for basic cost of living, volunteers are not given any supplies. Financial support for volunteers is extremely limited. The volunteers are provided with $160 a month, and the accommodation is provided by the hosting local community. If the local community does not provide accommodation, then Rwandan Francs ($30-40) is provided. $275 is put aside every month as relocation adjustment fees and reimbursed once the volunteer returns to the US. Misc. Support $2,500 for supplies over a two-year period. Volunteers are dispatched to local projects after 6 months and they receive maximum $50,000 financial support for the project. Volunteers are there as human resources with the purpose of technical support (skill transfer), therefore individual volunteers are not provided with supplies or financial support for projects. $10,000 is distributed every month as part of the Small Grants Program, where volunteers implement mini-projects with the local community. This financial support is provided by: USAID (Small Project Assistance Program, Feed the Future, Global Education Framework), State Department's eenergy and Climate Partnership of the Americas), President's Emergency Relief Program for AIDS, Volunteer Activities Supporting and Training, (VAST), and the Peace Corps Partnership Program (Civil sector funding). Some VAST programs liable for additional funding The local community funds are required to fund at least 25% of the total project costs. Implementation Situation of KOICA WFK 53

62 The need for mental health counselors to assist with the psychological problems that volunteers and their managers encounter, has been raised. There have also been requests to recruit and hire local security experts. Additionally volunteers have asked for experts on local projects to facilitate mentoring and consultation, as well as improved ex-post project management. But ultimately, as <Interview 5> shows, safety problems are very serious and the solution would be to place appropriate managerial staff in the right places, and effective division of labor among the managers. <Interview 5> The solution is not simply to increase the number of managers. I managed 30 people, including the volunteers. But I don't feel that work efficiency increases just by hiring more managers. Placement according to the type of work is very important, so if the division of labor was even a bit more efficient... Well, it is problem if one manager is managing 50 people, but if they are divided effectively, then it could be workable. I don't think that increasing the number of managers makes the project better. (J, manager). The adequacy of safety management system was scored 3.81, the administrative support system 4.12, and the cooperation from local cooperating agencies was This shows that the volunteer's perception was generally positive. However, the cooperation from local cooperating agencies and personnel was scored relatively low. When asked what was needed for effective safety management, the results were: "Hiring a safety expert" (47%), "Simplifying KOICA's administrative procedures for safety management" (20%), "Systematic emergency contact network" (17%), "Preemptive safety management training" (16%). The most frequent requests for improvements for effective volunteer work on the ground were: "Consistent communication with cooperating local agencies" (42%), "Feedback on volunteer work and local projects" (23%) and "Simplification of KOICA's administrative procedures" (23%). 54 Comprehensive Evaluation of KOICA WFK Program

63 e. Post-project Management Post-project management policies for volunteers include relocation fees once the volunteers return, scholarships, employment support through the job center, preferential treatment when volunteers apply for KOICA jobs and extensive hiring of volunteers as managers. Additionally, there are opportunities for startup support through the Small and Medium Business Administration, receiving certifications and recommendations and publishing one's volunteer experience stories. The KOICA ODA Training Center also works with returned volunteers to train international development cooperation experts and lecturers. They use the volunteer's experience to foster ODA experts. However, more customized employment support is needed. There also needs to be a counseling program to alleviate the depression and anxiety from returning after two years. The adequacy of post-project management scored 3.05 among the surveyed volunteers. This was the lowest score among all the WFK implementation systems broken down by the evaluation team. The volunteers perceived post-project management in a negative light, and especially the volunteers who were considering gaining employment (those who had undergraduate degrees or were attending graduate school). [Table 3-3] Adequacy of Post-Project Management Category N Average Adequacy of Post-Project Management Standard Deviation Implementation Situation of KOICA WFK 55

64 [Figure 3-7] Recommendations for post-project management The most important improvements that needed to made according to the returned volunteers were: "Customized employment information and job consulting" (53%), "Psychological counseling" (22%), "More active community for returned volunteers" (16%) f. Monitoring and Evaluation Evaluation on activities is conducted through on-ground evaluation conferences and multidimensional evaluations. Every year the head of the host agency holds a conference where the volunteers are evaluated According to interviews with volunteers from the 2012 Comprehensive Evaluation, there is general feedback going on through activity reports that volunteers submit, but the requests or recommendations that they write in the reports are not acted on immediately. Usually the reports are used by local cooperating and implementing agencies to evaluate volunteers and determine whether to renew their contracts or not. In <Interview 6> the volunteers said that the offices needed to monitor and give feedback on volunteer activities. <Interview 6> "I do all the activities by myself, so I don't know whether I am doing the right thing or not... there is no set curriculum. I expected 56 Comprehensive Evaluation of KOICA WFK Program

65 to teach (number) grade, but when I went there the students they were not on that level. I constantly wondered whether I was doing things correctly or not, and I did a local project a year after... I wish KOICA had done a mid-evaluation, but it didn't. I wish I had a Korean to talk to, but there was only reporting through documents"(c, volunteer) Yearly comprehensive satisfaction evaluations are conducted for the KOICA WFK. However there is not a results management system in place for the higher goals of the project. Yearly evaluations assess the volunteers and cooperating agencies, but there are no clear evaluation criteria and it is focused mainly on satisfaction surveys, as well as limits to overall results management and evaluation. Category Yearly Evaluation Comprehensive Evaluation Satisfaction Evaluation [Table 3-4] KOICA WFK Evaluation Description Evaluation for project results per country is conducted by KOICA offices abroad and submitted to KOICA HQ Cooperating Agencies: Conference with agency heads, Survey on project results Volunteers: Local evaluation conference, Survey on Project results Country project results are compiled and analyzed by HQ and published as yearly evaluation reports. Comprehensive evaluations are conducted by KOICA based on 3 yearsʼ worth of accumulated yearly evaluations. Recommendations from the evaluation are incorporated into future project strategies. Online surveys on the WFK is conducted every year, targeting volunteers and local cooperating agencies. Source: KOICA International Volunteer Project ( ) Comprehensive Evaluation Report. Since 2012, there has been a comprehensive satisfaction level survey on the overall WFK. The survey has used unified evaluation goals, evaluators and evaluation criteria since the Prime Minister's Office recommended it do so in its 2012 "Recommendation for Improving WFK Evaluation". Unified goals, evaluators Implementation Situation of KOICA WFK 57

66 and evaluation criteria have been used with consensus among WFK implementing agencies. In 2012, the evaluation only assessed volunteers, but was expanded in 2013 to include cooperating agencies. However without a theory of change model or evaluation framework, the sole use of quantitative surveys (satisfaction level surveys) cannot prove any causal links in the evaluation. Therefore, a systematic evaluation which uses a multidimensional evaluation framework that clarifies the evaluation categories, methods and indictors, with a clear evaluation targets is required. [Box] Monitoring and Evaluation Systems in Rwanda/Volunteer Agencies (JICA) JICA was given a recommendation that it should submit a report about its activities to cooperating agencies on a monthly basis. The volunteers submit a mid-term report a year after their work, and a final report presentation on their activities, lessons learned, failures and limitations 2 years after their work. The JICA local office head and volunteer coordinator evaluates the individual candidates two years after their activities have finished, using a one-page JOCV evaluation-form. The main evaluation criterion not only checks the results of their work, but also evaluates their attitude so that it can be used in the future for the JICA hiring process. However, they do not collect specific information about the results produced by the volunteers, nor do they require it. (Peace Corps) Concerning the primary goal of Peace Corps (technical assistance for education and public health), the volunteers enter data on a common form. The local office collects the data that the volunteers submit through online and send it to Washington DC. Most of the questions ask for written responses. The Department of Health and the Department of Education have MOUs and country agreements through which they collect data. However, there are limits to data collection methods like the one Peace Corps uses: 1) It is hard to measure results accurately (for example, it is hard to measure how the volunteers' results have affected the language abilities of students, because in order to evaluate this, one has to consider the variables and actors involved. 2) The validity of the data is considered to be a limitation to measuring project results. KOICA is using an "e-kov" system like the one in the [Box] below to submit reports. E-kov has been used since 2010 to share information such as personal information, activity reports and living environment. However, during the on-site 58 Comprehensive Evaluation of KOICA WFK Program

67 inspection in Rwanda, it was found that the lack of an adequate internet connection and absence of relevant information were making it hard for volunteers to use the system. It was also found that the KOICA Rwanda office needed to do monitoring and feedback evaluations. In Cambodia, an official cooperation system between cooperating agencies that would support volunteer activities and results was deemed necessary, even before a monitoring and evaluation process. [Box] KOICA WFK e-kov Administrative Management System (On-ground) Recommend that all volunteer reports and requests for service go through e-kov system. Try to conduct regular system training and receive remote support from HQ. (Korea) Collect statistics for audits, such as applicant statistics and volunteer statistics. Try to synchronize the system with the "ODA Integrated Information System" when it is established. (Data on activities) Reorganize and correct errors in the system concerning basic information on the volunteers. Improve the reliability of information on the integrated system and the volunteer website by analyzing dispatch periods and entering them on the system. Source: WFK Comprehensive Planning TF 년도 WFK Comprehensive Project Plan. Implementation Situation of KOICA WFK 59

68

69 Ⅳ. Examination of KOICA WFK Results 1. Volunteers 2. Hosting Country (Improved Standard of Living) 3. Sending Country (Strengthening Ties with Hosting Country)

70

71 Chapter Ⅳ Examination of KOICA WFK Results 1. Volunteers This evaluation organizes the results of the KOICA WFK into various levels (Goal, Outcome, Output) and shows the logical connections between them. For the Output and Outcome, the evaluation team conducted surveys and interviewed returned volunteers from Cambodia and Rwanda, who were active between 2012 and [Figure 4-1] Theory of Change: Volunteers Examination of KOICA WFK Results 63

72 For volunteers, the Output level consists of 7 elements for individual capacity building - Improved foreign language skills, increased work experience, increased expertise in the field, enhanced leadership skills, enhanced problem-solving skills, increased understanding of hosting country and region, increased understanding of development cooperation and 5 elements for cultivation of character - Increased open-mindedness, increased independence, increased self-efficacy, increased selflessness and clarification of life direction. The 3 elements for establishing a global network were - establishing a network with local residents, establishing a network with foreign volunteers and establishing a network among the (WFK) volunteers. [Table 4-1] Results per Level: Volunteer Goal Outcome Methodology Output Methodology Foreign language skills Work experience Expertise in the field Contribute to nurturing global talent Capacity building for volunteers Cultivate the character of volunteers Target: Volunteers Method: Surveys (Scale of 5 and Criticality), Interviews Leadership skills Problem-solving skills Knowledge on hosting country International cooperation knowledge Open-mindedness Independence Self-efficacy Selflessness Life direction Target: Volunteers Method: Surveys (Scale of 5 and Criticality), Interviews Establish a global network Network with locals Network with other volunteers Network among volunteers 64 Comprehensive Evaluation of KOICA WFK Program

73 After the analyzing the outputs of the capacity building of volunteers, it was found that the volunteers believed their individual capacities were strengthened through volunteer work. Volunteers ranked "Understanding of host region and country" (4.49) their greatest improvement. "Increased expertise in the field" (3.2), "Increased work experience" (3.43), "Enhanced English skills" (3.37) were ranked relatively low. When asked what they thought was most important, 31% of the respondents answered "Increasing expertise in the field" and "Increasing work experience". When their knowledge and experience in their field of expertise is enhanced consistently, it is then possible to deduce whether volunteers feel their capacity has been strengthened. [Table 4-2] Survey Results: Volunteers' Perception of Capacity (Building) Category N Average Standard Deviation English language skills Local language skills Work experience Expertise in Field Leadership Problem-Solving Skills Understanding of Region/Country Knowledge of Dev. Cooperation Examination of KOICA WFK Results 65

74 Most of the volunteers perceived themselves as having cultivated their character and personal development through the volunteer work. "Increased open-mindedness" scored 4.28, "Increased independence" (4.22), "Clarification of life direction" (3.77) and "Increased self-efficacy" (3.91) were scored relatively low. In the on-site inspection in Cambodia, most volunteers said that interacting with local residents helped to develop an open mind towards foreign people and cultures. Volunteers also said that after their volunteer projects, they perceived foreign laborers in Korea as equal partners and stopped having prejudices toward Cambodians. [Table 4-3] Survey Results: Volunteers' Perception of Character Cultivation Category N Average Standard Deviation Open-mindedness Independence Self-efficacy Selflessness Life Direction The volunteers thought that "Increased Open-mindedness" (58%) and "Increased selflessness" (25%), which is the willingness to help others, were the most important elements in character cultivation. 66 Comprehensive Evaluation of KOICA WFK Program

75 The 3 elements in building global networks were: building a network with local residents, building a network with foreign volunteers and building a network among the volunteers. The average score was 3.38, with the local network scoring the highest at 3.79 and the network with foreign volunteers scoring the lowest (3.06). On-site inspections in Cambodia found that networks with local residents that were functioning actively. Networks with other aid agencies differed from region and field of work, however generally they were not very active. As for networks amongst volunteers, because of the sudden increase in volunteers recently, interaction among them was not as lively as before. [Table 4-4] Survey Results: Volunteers' Perception of Global Networks Category N Average Standard Deviation Network with local residents Network with foreign volunteers Network among volunteers The volunteers thought that establishing a network with cooperating agencies and local community residents was by far the most important (68%). Networks with foreign volunteers and networks among WFK volunteers were 15% and 14% respectively, meaning that they were considered less important. Examination of KOICA WFK Results 67

76 At the Outcome level, capacity building for individual volunteers, cultivation of character and the establishment of a global network was achieved. Capacity building was scored the highest, whereas establishing a global network was scored relatively lower. Among the three result elements at the Outcome level, volunteers said that capacity building was the most important for contributing to the nurturing of global leaders. [Table 4-5] Survey Results: Volunteers' Perception of Results at Outcome Level Category N Average Standard Deviation Capacity building Cultivation of character Establishing a global network As shown in the table below, different perceptions of the project implementation system lead to difference perceptions of project results. Among the specific goals, capacity building was perceived differently from the perspective of education, safety, work management, and post-project management. Establishing a global network was also perceived differently according to demand assessment and training. This shows that in order to establish a global network, volunteers need to react positively to demand 68 Comprehensive Evaluation of KOICA WFK Program

77 assessment results. From the volunteer's perspective, the more positive the perception of implementation systems, the better the results and achievement rates of the volunteers were. As for the capacity of volunteers, the more positive the perception of training and post-project management was, the higher the achievement rate of results were. A similar relationship was found between demand assessment and training and achievement of results, from a global networks perspective. Contribut e to nurturing global talent [Table 4-6] Differences in Perception of Results According to Perception of Implementation Systems: Overall Capacity building Cultivation of character Establishing a global network Demand Assessment Recruitment & Hiring Training Activity Support P-M Management Per N Avg F P N Avg F P N Avg F P N Avg F P N Avg F P Neg Pos Total Neg Pos Total Neg Pos Total Hosting Country (Improved Standard of Living) This evaluation organizes the assessment of the KOICA WFK into various levels (Goal, Outcome, and Output) and shows the logical connections between them. For the Output and Outcome, the evaluation team conducted surveys and interviewed returned volunteers from Cambodia and Rwanda, who were active between 2012 and Local cooperating agency personnel who collaborated with volunteers in both countries were also surveyed and interviewed. Examination of KOICA WFK Results 69

78 [Figure 4-2] Theory of Change: Hosting Country The Output level results are made up of 2 elements (increased expertise on the volunteer's country region and nurturing prepared leaders) in creating regional experts, and 3 elements (increasing understanding of the activities by agencies volunteers are dispatched to, improving project sustainability and improving Korea's image) for improving the image of the country and agency that volunteers are dispatched to. 70 Comprehensive Evaluation of KOICA WFK Program

79 Improving the living standards of the residents [Table 4-7] Results per Level: Hosting Country Goal Outcome Methodology Output Methodology Increase of Human services use in public Material services services (Facilities and materials) Improved capacity of hosting agencies Improved policies for hosting agencies Target: Volunteers Method: Surveys (Scale of 5 and Criticality), Interviews Educational material Educational training Participation in training Administrative system Policy consulting Target: Volunteers currently active in Rwanda and Cambodia Method: Surveys (Scale of 5 and Criticality), Interviews To determine the results on the Output level and gather recommendations for collecting further results, the Output result indicators were used to understand the nature of the collected data. However, there is a lack of data collected , therefore the results were accumulated from the records of six volunteers who are currently active. Therefore, rather than being representative of the actual results, there is more emphasis on their role as examples for future examinations of results. Examination of KOICA WFK Results 71

80 [Table 4-8] Result Indicators for Outcomes from Hosting County Category Indicator Explanation Individual beneficiaries Local cooperating agencies Policies Average of number of beneficiaries (local residents) who receive public services (education, public health, agricultural development, industrial energy) from volunteers in a month Number and type of facilities that have been built or renovated during the time frame of volunteer activities Number and type of supplies that have provided during the time frame of volunteer activities Number and type of capacity building educational training that has been conducted for host agency personnel during the time frame of volunteer activities Number and type of educational content on capacity building that was created or developed for hosting agency personnel Number of host agency personnel who have participated in the capacity building educational training Number and type of administrative systems (forms) used by the host agency that were developed or improved Number of reports about improvements (related to research or policies) that were drafted ʻʻPublic servicesʼʼ means school classes, public health training and services, agricultural and regional development and etc. that come from volunteers. ʻʻBeneficiaryʼʼ means students, patients and town residents who interact directly with volunteers and receive public services from the agencies where dispatched volunteers work. (People who receive more than one service are still counted as one person). Eg: 20 students who receive 8 classes a month are considered as 20 beneficiaries (on average per month). ʻFacilitiesʼ means structures and buildings such as construction science labs, sports halls, computer labs, wells, washrooms. ʻSuppliesʼ means small stationary such as writing tools and document files, as well equipment that has been provided to local agencies and beneficiaries (computers, beam projectors and etc.). ʻCapacity building educational trainingsʼ mean workshops, training that been conducted directly and educational curriculum that been developed in person. ʻEducational content on capacity buildingʼ means all developed educational material, aids and tools. ʻNumber of participantsʼ means the number of participants from the host country who participated in workshops and trainings that were developed. ʻAdministrative systemsʼ means the administrative forms for activity reports and budgets that were developed or improved. ʻReports for policy improvementsʼ means the reports were about policy improvements for host agencies. 72 Comprehensive Evaluation of KOICA WFK Program

81 For the provision of human/material resources at the Output level, the approximate number of people who had received public services by KOICA WFK volunteers (dispatched in 2014) was also analysed. For Cambodia, an average of persons received such services and for Rwanda is was persons. There were a large number of beneficiaries who had received education-related public services from the volunteers. As for material resources, support increased 6 times for Cambodia and 10 times for Rwanda in 2012 (total of 16) to total 9 times for Cambodia and 12 times for Rwanda in 2013 (total of 21). The budget has increased as well, and the material for activities increased more than double, leading to more volunteers applying. The most supported type of local projects by both host and distributing countries were those related to building educational facilities and local project work in the education sector. [Figure 4-3] Percentage of Types of Local Project Support( ) Cambodia Rwanda [Figure 4-4] Percentage of Areas of Local Project Support( ) Cambodia Rwanda Examination of KOICA WFK Results 73

82 The numbers of capacity building educational training sessions by volunteers (dispatched in 2014) were an average of 3.4 for Cambodia and an average of 3.5 for Rwanda. Through the whole volunteer placement time frame, it is estimated that the numbers were 8.4 and 7.7 for Cambodia and Rwanda respectively. The number of beneficiaries of the programs was an average 54.2 persons for Cambodia and 56.8 persons for Rwanda. Most of participants of the capacity building educational training were students and teachers from schools. Therefore it can be concluded that the people related to the schools (students and teachers) are the ones whose capacities were most strengthened through volunteer work. [Table 4-9] Number of educational training sessions and number of participants Average per Country Category Category No. of Educational Trainings Developed No. of Participants in Educational Training Teacher training, classes, events (Korean Speech Content, Public Health Day, debates), teaching volunteers, volunteer efforts No. of Developed Capacity Building Contents Educational material for schools and teachers. Class materials and posters Cambodia In <Interview 18> the host agency personnel were found to be aware of the importance of the individual employees and the agency as a whole. They hoped that the volunteers share their expert knowledge in order to do capacity building for the agency and its personnel, while acting independently with their own plan of action. <Interview 18> But our aim, as my school, we don't ask the volunteers for their projects. We prevent the projects. We ask the volunteers just to help in practical, teaching, and learning process. Not the projects. The project has 74 Comprehensive Evaluation of KOICA WFK Program

83 come, when we were discussing about our plan for our action plan of our strategies plans and then they come in to help. (M, employee at hosting agency) Evidence of improvement to the administrative systems and reports were barely visible, with 0.8 cases on average for Cambodia and 1 case on average for Rwanda. The reason for this could be that the volunteers have a different role compared to the mid-to-long term advisors, and therefore there is a low level of progress on improving policies. At the Outcome Level for the host country, which was evaluated through personnel from local cooperating agencies, an average score of 4.16 was given for providing public services, capacity building for host agencies and improving host agency policies. The highest score was 4.45 for capacity building for collaborating agencies. Therefore it can be said that volunteer work is mainly conducted around cooperation activities with local cooperating agencies, rather than activities that directly affect the local residents or improve host agency policies. [Table 4-10] Survey Results: Cooperating agency personnel on results at the Outcome level Category N Average Standard Deviation Public services Capacity building Improving policies Examination of KOICA WFK Results 75

84 When the same survey was given to volunteers, it scored an average of 3.48, which was lower than the score given by the host country. Increasing public services and capacity building had higher scores than improving host agency policies. [Table 4-11] Survey Results: Volunteers on results at the Outcome level Category N Average Standard Deviation Public services Capacity building Improving policies In regards to capacity building for host agencies, there are differences in the perception of the results according to different views on demand assessment, recruitment and selection, training, service and safety management are conducted. Policy improvement is influenced by recruitment and selection, training and activity support. Perceptions of how adequate the recruitment, selection and training is, acts as an important variable to improving the lives of local residents. 76 Comprehensive Evaluation of KOICA WFK Program

85 When asked which element had the greatest influence on improving lives, 79% of the local agency personnel and 40% of volunteers answered that capacity building was the important. In Cambodia, interviews with host agencies showed that both the volunteers and host agencies were strengthening their capacity, but highlighted the absence of official educational training for knowledge transfer. Most methods relied on indirect methods such as observation. [Table 4-12] Differences in Perception of Results According to Perception of Implementation Systems: Overall Demand Assessment Recruitment & Hiring Training Activity Support P-M Management Per N Avg F P N Avg F P N Avg F P N Avg F P N Avg F P Neg Capacity Pos building Total Contribut Neg e to Cultivation nurturing of Pos global character talent Total Neg Establishing a global network Pos Total Examination of KOICA WFK Results 77

86 3. Sending Country (Strengthening Ties with Hosting Country) Survey results and data collected from evaluations was used to measure results on the Output level, and survey results from local cooperating agencies were used in measuring Outcome level results. [Figure 4-5] Theory of Change: Sending Country The Output level consists of 2 elements for nurturing regional experts (increasing regional and national expertise of the volunteers and nurturing experts who have completed volunteer service) and 3 elements for improving the image of the distributing country and agency (increased understanding of KOICA activities, improved project sustainability in relation to KOICA and improved image of Korea). 78 Comprehensive Evaluation of KOICA WFK Program

87 Enhance relationships between hosting and distributing country [Table 4-13] Results per Level: Sending Country Goal Outcome Methodology Output Methodology Target: Local cooperating agencies Increased regional/country expertise of volunteers Increased network of regional experts Enhanced image of distributing country & KOICA Target: Local cooperating agencies Method: Survey (Scale of 5) Nurturing experts who have completed volunteer service: % of volunteers who have completed their service Increased understanding of KOICA activities Increased project sustainability (in relation to KOICA) Improved image of Korea Method: Survey (Scale of 5), Interview Method: Data Calculation Target: Local cooperating agencies Method: Survey (Scale of 5), Interview The average score for an increased network of regional experts was 4.38, meaning that most local agency personnel felt that volunteer work increased the level of expertise concerning the region and country. The indicator "Nurturing experts who have completed volunteer service" was calculated by determining the percentage of volunteers who had completed their volunteer service (the years which would have influenced the project results from 2012 to 2014), 88.1% of volunteers in Cambodia and 76.2% of volunteers in Rwanda completed their service. Therefore, it can be said that two-thirds of the volunteers who had been dispatched had expertise on the region and country they worked in. How to effectively use these experts with regional expertise is a matter that needs to be discussed. Examination of KOICA WFK Results 79

88 Most local cooperating agency personnel indicated that KOICA and Korea's image improved through the volunteer work, giving it an average score of 4.6. Between 2012 to 2014, the number of Rwandan agencies that had asked for volunteers the next year had increased, and in the same period there was a high number of volunteers in Cambodia, (apart from in 2013). This also means that project sustainability in relation to KOICA was also high in general. The reason for this is assumed to be Korea's enhanced image from the perspective of the host agencies. Upon verifying the Outcome level results from host agencies, an increased network of regional experts and an improvement in the image of distributing country and agency scored well - an average of 4.6 (host agency aspect) and 3.93 (volunteer aspect). When asked what was the most important factor in enhancing relationships between and the host and distributing countries, 31% answered increased project sustainability in relation to KOICA and increased understanding. [Table 4-14] Enhance relationships between hosting and sending country: Cooperating agency Category N Average Standard Deviation Understanding of activities Project sustainability Improved image of Korea Comprehensive Evaluation of KOICA WFK Program