EVALUATION OF NETCORPS CANADA INTERNATIONAL PROGRAM (NCI)

Transcription

1 EVALUATION OF NETCORPS CANADA INTERNATIONAL PROGRAM (NCI) EVALUATION DIVISION PERFORMANCE AND KNOWLEDGE MANAGEMENT BRANCH MAY 2007 Canadian International Development Agency 200 Promenade du Portage Gatineau, Quebec K1A 0G4 Tel: (819) Toll free: Fax: (819) (For the hearing and speech impaired only (TDD/TTY): (819) Toll free for the hearing and speech impaired only: )

2

3 ACKNOWLEDGEMENTS The Evaluation Division would like to express its gratitude to the representatives of the recipient country host organizations for their collaboration during the evaluation team s field visit in Kenya, Burkina Faso, Niger and Ghana. Their inputs and video testimonies with regard to the development results of this Program were crucial to achieving the prime objective of this evaluation. The support of the NetCorps Coalition members was also much appreciated. Special thanks to Jetske Duintjer and Sharmaine Nelles who provided detailed documentation to the evaluators throughout the evaluation process and helped to organize the field visits. The advice provided by the NetCorps Steering Committee was valuable and provided added value to the evaluation process and report. Thanks to all the members. Special thanks to the client Branch, the Office for Democratic Governance, who requested this evaluation and provided their collaboration throughout the evaluation, i.e. Mark Foley and Glenn Brigaldino. Canadian Partnership Branch representatives, Lillian Chaterjee and Patricia Zakaib, also provided valuable information and collaboration. Finally, the Evaluation Division also acknowledges the support of the evaluation team members - Louise Lapointe for her participation in the field mission, interviews and benchmarking exercise, and Larry Thériault, who managed this evaluation and prepared this report. Goberdhan Singh Director, Evaluation Division i

4 ii

5 TABLE OF CONTENTS Acknowledgements...i Acronyms...v EXECUTIVE SUMMARY... vii 1. INTRODUCTION PROGRAM PROFILE Background of the Program Implementing Organization Targeted Beneficiaries Program Objectives Program Budget and Expenditures EVALUATION PROFILE Purpose of the Evaluation Key Audiences - Stakeholders Coverage and Sampling Data Collection Sources Video Testimonies Evaluation Framework & Methodology EVALUATION FINDINGS Efficiency and Effectiveness Satisfaction with Program Relevance Other Findings Benchmarking Previous evaluations of NetCorps International Program Relevance and fit of NetCorps to CIDA s priorities Relation with CIDA s Volunteer Cooperation Program CONCLUSIONS RECOMMENDATIONS General recommendations Pre-departure recommendations Recommendations for the Implementing Agencies Recommendations for CIDA management LESSONS APPENDICES...27 Appendix I: List of participating host organizations from the field...27 Appendix II: List of interviewees...28 Evaluation of the NetCorps Canada International Program (NCI) iii

6 Appendix III: Management Response by Oxfam-Québec...30 Appendix IV: Management Response by CIDA...31 Appendix V: Bibliography...37 List of Tables Table 1: NetCorps Coalition Members Canadian NGOs 1 Table 2: Logic Framework Analysis 3 Table 3: Contribution of Industry Canada (IC) to NetCorps 4 Table 4: Profile of the 32 Host organizations selected for field visits 6 Table 5: Data Collection Framework 8 Table 6: CIDA s priority sector and high level development results indicators 9 Table 7: Development results in sampled organizations based on output and outcome indicators 10 Table 8: Development results according to Host organizations, NetCorps volunteers and the evaluation team 12 Table 9: Benchmarking with other ICT volunteer groups 17 Table 10: Comparison between NetCorps and IYIP in Table 11: CIDA s Contribution Agreements with NetCorps Coalition Members 22 Table 12: Options for the Future of NetCorps 23 Evaluation of the NetCorps Canada International Program (NCI) iv

7 ACRONYMS BOG CC CCI CIDA CPB CSIH CWY DAC DFAIT DFI DOT EC HRI HRSDC IC ICT ICT4D ICV IDRC IISD IYIP LFA MDG NCI NGO OECD OfDG PKMB PRAS TOR VCP VSO VSP YES YI Business Operation Group Canada Corps Canadian Crossroads International Canadian International Development Agency Canadian Partnership Branch Canadian Society for International Health Canada World Youth Development Aid Committee Department of Foreign Affairs and International Trade Digital Freedom Initiative Digital Opportunity Trust Evaluation Committee Human Rights Internet Human Resources and Skills Development Canada Industry Canada Information and Communication Technologies Information and Communication Technologies for Development International Conference Volunteer International Development Research Centre International Institute for Sustainable Development International Youth Internship Program Logical Framework Analysis Millennium Development Goals NetCorps Canada International Non-governmental Organization Organization for Economic and Co-operation Development Office for Democratic Governance Performance Knowledge Management Branch Programming, Reporting and Accountability Structure Terms of Reference Voluntary Cooperation Program Voluntary Service Overseas Volunteer Sector Program Youth Employment Strategy Youth Internship Evaluation of the NetCorps Canada International Program (NCI) v

8 Evaluation of the NetCorps Canada International Program (NCI) vi

9 EXECUTIVE SUMMARY Background The NetCorps Canada International (NCI) Program was launched in 1997 as a joint initiative of Industry Canada and the Government of Canada s interdepartmental Youth Employment Strategy (YES). Its primary purpose was to provide university graduates (aged 19-30) with employment experience in the field of ICT (Information and Communication Technologies). The program was implemented by a coalition of nine Canadian NGOs with experience in international volunteer sending (see Table 1). NetCorps interns were placed with developing country partners for a period of 6-9 months to provide ICT support that would strengthen the capacity of the host organization. Since employability was the primary focus of the program, results for developing country partners were not systematically documented. When Industry Canada decided not to support the NetCorps program beyond FY 2006/07, the NetCorps Coalition approached CIDA to consider funding the program. All of the Coalition members have a funding relationship with CIDA for volunteer sending activities. Purpose of the Evaluation CIDA s Office for Democratic Governance requested an evaluation of the NetCorps program to assess its developmental relevance, effectiveness and sustainability as a mechanism/programming approach for supporting ICT for development. The evaluation findings will facilitate CIDA s decision-making concerning funding of a NetCorps-type of project for a period of 5 years at approximately $5 million per year (approximate total $25 million). Summary of Findings The evaluation confirmed the relevance, efficiency and effectiveness of the Net- Corps International Program. Field visits confirmed that NetCorps has had a beneficial impact on host organizations in terms of enabling results i.e. improvements in the efficiency and effectiveness of their management, more transparency and accountability of their operations, capacity building and women empowerment. Skills transfer and sustainability of results were also confirmed. Feedback from host organizations indicates a high level of satisfaction with the NCI Program. The NetCorps program is consistent with CIDA s approach to ICT as a strategy to support development. While ICT is considered a crucial tool to achieve development results in the areas of good governance and capacity building, the Agency didn t have, during the evaluation period, define guidelines to integrate systematically the ICT dimension in its programming. CIDA has supported ICT initiatives through the Canada Fund for Africa (ending in 2008) but no other programs are dedicated specifically to ICT. The satisfaction of host organizations with the NetCorps internships demonstrates the niche for an ICT voluntary program. NetCorps Coalition members have demonstrated their capacity to deliver an ICT program through youth and volunteers using their existing field presence/ infrastructure. However, improvements are needed in the design Evaluation of the NetCorps Canada International Program (NCI) vii

10 of the program in order to reinforce development results. Conclusions The evaluation team was asked to develop options for future funding of the program for consideration by CIDA senior management. Four options have been presented (see Table 11). The evaluators view is that the fourth option is the most viable approach to ensuring the continuity of the program, which has produced development results, and has the potential for longer-term impact. Recommendations 1. General recommendations 1.1 The eligibility criteria and selection process for interns should be strengthened to put more emphasis on experience and education in ICT and training skills that will meet the host organization s needs. Maturity and cultural sensitivity should also be considered. 1.2 The length of placement should be established in consultation with the host organization and reflected in the TOR to ensure that the organization s needs will be met, while recognizing that some period is needed for the intern to adapt to the setting (depending on the intern s previous experience and the field environment constraints). 2. Pre-departure recommendations 2.1 Contacts between the intern and the host organization prior to departure should be increased in order to ensure a common understanding of the TOR and the needs and constraints of the host organization s environment. 2.2 Interns should be trained as trainers and equipped with appropriate ICT material (diagnostic tools, free software, testing equipment). In addition, provision should be made to allow for local purchasing of equipment where ICT services are available. 3. Recommendations for the coalition of Canadian NGOs 3.1 In order to comply with CIDA s requirements to demonstrate development results to the Canadian Public, the coalition members must have an adequate MfDR (Management for Development Results) framework to ensure adequate reporting on results and for evidence-based learning. 3.2 As reflected in the Coalition Members Internal Reflection document 1, implementing agencies need to coordinate their efforts in country and work jointly on projects wherever possible to increase the potential for synergy (e.g. monitoring of projects in the field). 3.3 The criteria and processes used to select host organizations should be reviewed to ensure an appropriate match between the internship and the needs of the host organization, and effective use of program funds. 3.4 Systematic follow-up with interns and host organizations throughout and after placements, and 1 NetCorps Canada International. An Internal Reflection to Maximize the Benefits of the Coalition, Heather Creech, IISD, March Evaluation of the NetCorps Canada International Program (NCI) viii

11 improved reporting of development results would contribute to enhancing the sustainability of results, as well as the transfer of knowledge and lessons learned. 4. Recommendations for CIDA management 4.1 That CIDA consider the fourth option for future funding of the NetCorps program presented by the evaluation team. The team believes that this option is the most viable to ensure the continuity of a Program, which has produced development results. 4.2 At the corporate level, CIDA, through its Business Operation Group (BOG) should define guidelines to integrate the ICT dimension in its programming since it s a crucial tool for development. 4.3 In the event that CIDA chooses to integrate the NetCorps program with other existing programming, budget allocations should be provided in a timely fashion. This would allow more lead-time to the implementing agencies for the selection of projects and interns as well as organizing the logistics for placements to the field. Evaluation of the NetCorps Canada International Program (NCI) ix

12

13 1. INTRODUCTION The evaluation of the NetCorps Program was undertaken by the Evaluation Division of CIDA s Performance and Knowledge Management Branch (PKMB) in collaboration with CIDA s Office for Democratic Governance in the fall of It provides input to CIDA senior management concerning: i) the efficiency of the NetCorps Program as an approach for supporting ICT for development efforts, ii) the relevance and the fit of this program with CIDA s medium to longer-term priorities, and iii) the best utilization of CIDA resources for a NetCorps-type program beyond 2006/ PROGRAM PROFILE 2.1 Background of the Program The NetCorps Canada International (NCI) Program is a joint initiative of Industry Canada s Connecting Canadians Strategy and the Government of Canada s interdepartmental Youth Employment Strategy (YES). Launched in 1997, it gives young people the opportunity to participate in innovative internships in the field of international development and ICT (Information and Communication Technologies). Since the Youth Employment Strategy focused on youth employability and Industry Canada has a national mandate, the NCI was not originally designed to focus on development cooperation. Producing results for developing country partners was a secondary objective and results were not systematically reported. 2.2 Implementing Organization In partnership with Industry Canada, the project is implemented by a coalition of some of Canada's largest international development organizations that sent volunteers abroad. As the NetCorps Coalition is a strategic alliance without legal status, Oxfam- Québec has been designated by the members of the Coalition as the lead agency to negotiate and sign contracts with funding partners. Table 1 shows the members of the NetCorps Coalition. Table 1: NetCorps Coalition members - Canadian NGOs Canada World Youth (CWY) International Institute for Sustainable Development (IISD) Alternatives OXFAM-Québec (Coalition lead) VSO Canada Canadian Crossroads International (CCI) Canadian Society for International Health (CSIH) Human Rights Internet (HRI) CUSO Evaluation of the NetCorps Canada International Program (NCI) 1

14 2.3 Targeted Beneficiaries NetCorps gives Canadian youth between 19 and 30, who hold a post-secondary degree, the opportunity to participate in an internship lasting from three to six months, including time both in Canada and overseas. The program annually gives some 210 young Canadians the opportunity to contribute their skills and knowledge to building the capacity of civil society and government organizations in developing countries. Between 1999 and 2007, NetCorps Canada International placed over 1900 participants in over 1000 host organizations and communities in less developed countries. Internships normally take place in Africa, Asia, Americas and the Caribbean, and Eastern Europe. 2.4 Program Objectives According to the NetCorps International official website, the key overall objectives of the program are: 1. To provide Canadian youth with advanced experience and capabilities related to the integration of ICT, thereby making them highly qualified, competitive, responsive and adaptable in the knowledge-based economy; 2. To support developing countries' efforts to enhance their information and communication technologies capacity; 3. To support developmental objectives of partners to improve their information and communication technologies capacity abroad in fields such as health, education and agriculture; 4. To profile Canadian leadership in the use of information technology for social and economic development; Objectives 2 and 3 of the NetCorps Program support the developmental component of NetCorps. These objectives are coherent with CIDA s draft Logical Framework prepared by the former Canada Corps in anticipation that they would run the Program eventually. They are as follows: to encourage the adoption and use of information and communications technologies (ICTs) within the public sector; to enhance citizen participation for more effective delivery of government programs and services; as well as to strengthen the governance, infrastructure, and ICT capacity of certain local public and Para public institutions and certain civil society organizations in a select number of ODA eligible countries. Table 2 identifies the developmental impact and long-term outcomes of the CIDA s Program Logical Framework Analysis (LFA) with key activities, outputs/outcomes and indicators. Evaluation of the NetCorps Canada International Program (NCI) 2

15 Table 2: Logic Framework Analysis Developmental Impacts / Outcomes and Indicators as prepared by former Canada Corps Post project impacts and long-term outcomes Improved good governance procedures and practices and enhanced organization capacity of selected civil society organizations and public and para-public institutions. KEY ACTIVITIES KEY OUTPUTS INDICATORS KEY OUTCOMES INDICATORS 1. Partner and placement selection, including women s organizations 2. Recruitment and selection of 258 volunteers (40% women) 3. Pre-departure orientation for volunteers Young Canadians working in different civil society, public and parapublic institutions and organizations; 5. Progress reports of ICT 1.1 Selected partners use new ICT related solutions to improve organization or to provide client populations with greater access to information 1.1 Number of overseas partners using ICT solutions provided through the program, including number of women s organizations and % of women s organizations in the total number of partners. 1.1 Satisfaction of partner organizations with ICT tools and solutions provided through the program 1.1 Number of organizations which received staff training including number of women s organizations New or improved ICT tools (such as websites, data collection systems and data sharing systems, etc.) Enhanced organizational capacity of selected civil society organizations and public/ para-public institutions to respond to the expressed needs of client populations. Greater access to information for citizens. Number of local public and para-public institutions and selected civil society organizations through support from volunteers; Progress reports document increase in organization capacity and/or improved access to information for clients in selected organizations and institutions 1.2 Staff members in selected partner org. are trained in the use of ICT tools, procedures and processes 1.2 Number of organizations which received staff training including number of women s organizations Evaluation of the NetCorps Canada International Program (NCI) 3

16 2.5 Program Budget and Expenditures Industry Canada contributed 30 million dollars to the NetCorps Program from its inception in 1999 to The average cost for the 1713 interns placed by the program was $17,176 per intern. Table 3 provides a summary of program expenditures. Table 3: Contribution of Industry Canada (IC) to NetCorps Number of interns Industry Canada Contribution Secretariat cost with common services * (thousand $) IC cost $ per intern ** (12%) 17, (15%) 17, (14%) 17, (14%) 17, (14 %) 17, (11%) 16, (10%) 16,663 TOTAL AVERAGE (13%) 17,176 *The secretariat costs & common services are related to the coalition secretariat of the 9 Canadian NGOs. As the NetCorps Coalition is a strategic alliance without legal status, Oxfam-Québec has been designated by the members of the Coalition as the lead agency to negotiate and sign contracts with funding partners. One of the reason for the Secretariat existence was to ensure that Industry Canada would have only one interlocutor to deal with and more importantly that this Unit or Secretariat would provide common services to the NetCorps Program and its interns such as orientation briefings sessions before departure, training, etc. **It should be noted that within the Industry Canada Contribution there was a fix 20% allocated to administration costs for each intern placement overseas. 2 CIDA has contributed an additional $ to the program Information and Communication Technologies (ICT4G) in that has not been accounted for in table 3. Evaluation of the NetCorps Canada International Program (NCI) 4

17 3. EVALUATION PROFILE 3.1 Purpose of the Evaluation The purpose of this evaluation is to inform CIDA of the developmental relevance, sustainability and effectiveness of the NetCorps program as a mechanism/programming approach for supporting ICT for development efforts. Such information will serve as an input to CIDA s internal decision-making processes aimed at determining the best utilization of CIDA resources to support ICT initiatives of this type (approximately $25 million over 5 years). This evaluation will provide advice and recommendations to CIDA management with regard to future support for a NetCorps-type project beyond 2006/ Key Audiences - Stakeholders The clients for this evaluation are CIDA s Evaluation Committee (EC) and CIDA s Office for Democratic Governance. The other main stakeholders are the members of the NetCorps Coalition, Industry Canada, CIDA s Partnership Branch, and eventually the Canadian Public once the report has been approved and published. 3.3 Coverage and Sampling Data for the evaluation was collected through site visits in East and West Africa between Nov 25th and Dec 10th These two regions were selected because of the high frequency of Program activities and the wide range of Coalition members projects. A total of 32 host organizations in Kenya, Ghana, Burkina Faso and Niger were interviewed. The projects surveyed covered the period. The sample is representative of the spectrum of CIDA s priority sectors, the different types of organizations, and the ICT interventions provided. Only one Host organization was eliminated from data collection because the internship had not begun at the time of the field visit. Twenty-six interns who provided ICT interventions for the Host organizations included in the sample were also interviewed. Table 4 provides information on the 32 Host organizations in the sample. Evaluation of the NetCorps Canada International Program (NCI) 5

18 Table 4: Profile of the 32 Host organizations selected for field visits INFORMATION ON THE 32 HOST ORGANIZATIONS N=32 INFORMATION PROFILE CATEGORY Country of origin of the 32 Host organizations sampled Portion % of sample related to corresponding information profile category Burkina Faso (40%) Kenya (28 %) Niger (16%) Ghana (16%) Area of Host organizations interventions * related to the 32 Host. Org. sampled Governance (71%) Education (68%) Health (26%) Gender (23%) Private sector Development (10%) Type of organization related to sample NGO community (68%) Public sector (13%) Part of UN umbrella (13%) Private Sector (6%) Type of ICT intervention requested *by the 32 org. sampled Computerinternet (100%) Printing / Media (19%) Radio (6%) Television (3%) Source of funding of 32 org. Mixed sources (42%) Fund Raising (35%) Statutory (10%) Outreach of 32 org. National (68%) Regional (13%) Local (13%) International (6%) *Some Host organizations intervened in more than on area ** Some Host organizations requested for ICT support in more than one area Evaluation of the NetCorps Canada International Program (NCI) 6

19 3.4 Data Collection Sources A variety of sources was used to ensure triangulation of data and validation of information, including: Document and literature review: Interviews: Documents from Coalition members and the Coalition Secretariat, including the document, An internal Reflection to Maximize the Benefits of the Coalition; Reports from Interns and Host organizations; CIDA literature: Trends in ICT4D and Policy Implications for CIDA (YED 2006), CIDA s Toolkit on how to Integrate ICT and Development Projects (December 2006), The Power of Volunteering A Review of the Canadian Volunteer Cooperation Program (March 2005). Host organizations: Face-to-face, videotaped interviews with the official representatives of 31 Host organizations in Kenya, Ghana, Burkina Faso and Niger; Coalition members: Face-to-face or telephone interviews with all 9 Coalition members in Canada; Face-to-face interviews with VSO and CUSO Ghana field representatives and Oxfam-Québec Burkina Faso representative; Interns: Telephone interviews with 26 interns who participated in internships with Host organizations visited between ; CIDA: Senior economic specialist, Policy Branch, ICT specialist, Asia Branch, Program manager and Director, Office for Democratic Governance, program managers, Director and Executive Director, Canadian Partnership Branch Benchmarking: In order to provide key information for decision-making the evaluation team undertook a benchmarking exercise, obtaining comparative information from the following recognized ICT organizations: GeekCorps: USA, NGO UNITeS: Germany, Agency within the United Nations promoting ICT volunteering Peace Corps: USA, NGO Trust for the Americas (previously NetCorps America): USA, NGO Evaluation of the NetCorps Canada International Program (NCI) 7

20 Table 5 summarizes the different sources of information used during the evaluation for each of the key areas covered (relevance, efficiency and effectiveness, sustainability, ICT fit and use within CIDA). Table 5: Data Collection Framework INFORMATION SOURCES AREAS COVERED Host org. Interns CIDA Coalition Benchmark INT Q INT INT LIT INT LIT INT LIT Relevance Efficiency & Effectiveness Sustainability Fit & Use with CIDA INT= Interviews Q= Questionnaires Lit= Literature (See bibliography Annex 5) 3.5 Video Testimonies During the field visit, the evaluation team decided to innovate its methodology to collect and communicate information related to development results. In order to facilitate a more direct delivery of messages from the beneficiaries of the NetCorps Program to CIDA senior management and the Canadian public, the evaluators used the video camera to record interviews. Short video clips (2 minutes average) of testimonies were prepared to highlight host organizations responses. Section provides a sample of these messages. 3.6 Evaluation Framework & Methodology In order to assess the ICT spin-offs in terms of development results, three main categories were established according to CIDA s priority sectors: a) Good Governance & Democracy, b) Capacity Building, and c) Gender. The evaluation team developed a first set of high level indicators (outcomes) and then specific shorter-term output indicators for the analysis of these categories. ICT is a tool or a mean to increase the management efficiency and effectiveness of the Host Organizations. For example, the existence of up-to-date financial statements and websites in the host organizations visited were an indication for the evaluation team of positive impact on the transparency & accountability of the recipient organizations. Another example is the spin offs of the Evaluation of the NetCorps Canada International Program (NCI) 8

21 ICT training provided to the Hosts organizations representatives who were able to train in return their members and the communities. The following table 6 summarizes the above-mentioned categories of indicators: Table 6: CIDA s priority sectors and high level development results indicators CIDA PRIORITY OUTCOME OUTPUT INDICATORS IN HOST ORG. Good Governance & Democracy Transparency and accountability Annual reports Financial statements Up to date accounting Websites Database & Data sharing Media information Templates for data collection Advocacy & Citizen participation Management Information System Capacity Building Local ownership of programs Network Staff trained & Trainers (training session provided) Up to date websites Up to date database Up to date data sharing Up to date templates for data collection Active Networking Advocacy Gender Specific Women s Empowerment Women s participation Women s training Women s access to information Women s advocacy Two sets of indicators were developed for each area, 21 indicators for outputs and 11 for outcomes. Using the field data collected and CIDA s Program LFA, the frequency of occurrence of these indicators was averaged to determine the level of achievement of CIDA s three priority sectors of development results. Table 7 on the following page outlines the detailed results in each category of development results, based on the findings from the field mission and reports from host organizations and interns. Indicators were also developed to assess: a) the sustainability of NetCorps interventions, and b) skills transfer to the recipient organizations. Evaluation of the NetCorps Canada International Program (NCI) 9

22 4. EVALUATION FINDINGS 4.1 Efficiency and Effectiveness Direct development results at the output and outcome levels Based on interviews with, and reports from, host organizations and interns, the evaluation team found that: 84% of host organizations confirmed development results (outputs and outcomes) from the interventions of NetCorps interns, which contributed to an increase in the efficiency and effectiveness of their operations. 73% of interns attributed development results to the host organizations as a result of their placements. The following table 7 outlines the results of our field mission in 4 countries i.e. the summary of findings in terms of development results based on 21 specific indicators at the output level and 11 at the outcome level. Table 7: Development results in sampled organizations based on output and outcome indicators n = number of host organizations sampled for which development results have been confirmed on site %= occurrence of results observed in the 31 host organizations visited I- Good governance and Democracy Outcome Improved Transparency/ Accountability Spin offs of ICT - Output indicators, which had a positive impact on enabling results of host organizations (e.g. improved management, operations and communication) n = % Annual reports % Financial statements 7 23 % Up to date accounting % Websites % Database % Data sharing % Media information % Templates for data collection % Advocacy % Citizen participation % Management Information System % Network % AVERAGE 70 % Evaluation of the NetCorps Canada International Program (NCI) 10

23 II- Capacity Building Outcome Spin offs of ICT - Output indicators n = % Local ownership of programs Training to staff % Train the trainers % Up to date websites % Up to date database % Up to date data sharing % Up to date templates for data collection % Active Networking % Advocacy % AVERAGE 90 % III- Gender specific Outcome Spin offs of ICT- Output indicators n = % Women s empowerment Women s participation % Women s training % Women s access to information % Women s advocacy % AVERAGE 66 % Table 7 demonstrates that there was significant evidence of ICT spin offs in terms of development results at the output and outcome levels in the areas of: transparency & accountability (72 %), which is part of Good Governance, and Local Ownership of Programs i.e. the Capacity Building (89%). With regard to the area of Gender, the random sample of 31 host organizations included 7 organizations specifically aimed at helping women in different fields such as family violence, women s rights, etc. These organizations represent 26% of the total sample. Specific results related to Gender i.e. women empowerment was achieved by all of these 7 organizations (100%), based on the output and outcome levels indicators outlined in Table Sustainability of results and skills transfer One of the main objectives of the evaluation was to document evidence of: a) the sustainability of NetCorps interventions, and b) skills transfer to the recipient organizations. a) Specific indicators developed for sustainability were: Presence of spin offs following NetCorps support (e.g. unexpected development benefits occurring after the internship); Web sites developed by the NetCorps volunteers maintained by the host organization staff after their departure; and Evidence that the tools developed by the intern (data base, software, networking, server, etc.) are still used by the host organization staff (e.g. Evaluation of the NetCorps Canada International Program (NCI) 11

24 using the software to prepare their activity and financial periodic financial statements.) b) Specific indicators related to skills transfer were: The ability of NetCorps volunteers to provide ICT training activities in an effective and efficient manner and to transfer their knowledge to host organizations members; The ability of the host organization staff who received training to communicate their knowledge to other members of their organization; The capacity of the host organization staff to maintain and repair systems put in place by interns; and The ability of the organization to identify further knowledge gaps in the area of ICTs. The results of the evaluation indicate that the NetCorps Program was successful in providing interventions that were sustainable and transferred knowledge to the recipient organizations. The evaluators confirmed evidence of sustainability of results in 81% of the cases and skills transfer in 94% of host organizations surveyed. Table 8 summarizes development results for the overall Program. Table 8: Development results according to Host organizations, NetCorps volunteers and the evaluation team Development Results according to interns & Hosts Satisfaction of Host Organizations surveyed Sustainability of development results according to evaluators Skills transfer according to evaluators Interns: Yes 73 % Very satisfied 61 % Yes 81 % Yes 94 % Interns: No 27 %* Satisfied 39 % No 19 % * No 6 % * Hosts: Yes 84 % Unsatisfied 0 % Hosts: No 16 %* * Reasons provided for non-satisfaction were related, for example, to not enough preparation of interns before departure, to a lack of ICT equipment and a too short period of time of the internship to allow the interns to complete their program. Areas for improvements and lessons learned based on our interviews and from various sources have been identified and documented in sub-sections 4.2 to 4.6. Evaluation of the NetCorps Canada International Program (NCI) 12

25 4.2 Satisfaction with Program The evaluation team collected information from the recipient organizations with respect to their overall degree of satisfaction with the NetCorps Program and the interns placements. All host organizations expressed satisfaction with the Program and the interns placements, with 61% indicating they were very satisfied and 39 % indicating they were satisfied. 3 The main reasons given for their high level of satisfaction were: their need for ICT support was significant, and the services provided by the NetCorps volunteers was in line with their needs; they couldn t afford the cost of a professional ICT consultant and the services provided by NetCorps volunteers was free; and NetCorps volunteers support provided long-term added value by improving the organization. These improvements are reflected in the findings shown in Table Relevance Host organizations generally indicated that NetCorps interventions were very relevant to their mandate and activities. The following sample comments are taken from the videotaped interviews collected during the field visits. 3 None of the host organizations reported being not satisfied. Evaluation of the NetCorps Canada International Program (NCI) 13

26 Sandra Cofie, LL.B, LL.M, Director Judicial Service of Ghana in efforts to make the judicial service of Ghana transparent and accountable. Ouiali Ouaka, Présidente CONGAFEN Coordination of 60 women associations throughout Niger NetCorps internship led to the opening of a new computer center for women in order to further their training. In this case, the NetCorps Volunteer helped the Ghana Justice Department to develop a database and a data collection methodology, which were used to produce the first annual reports of the Justice Department for the Parliament of Ghana and the public. Templates have been developed to facilitate data collection from the various court levels in the different regions of Ghana. This intervention had a positive impact on the ability of the Justice Department to be more transparent based on improved efficiency and effectiveness of the operations. The CONGAFEN is a key women s organization in Niger, which brings together more than 60 women s associations throughout the country. The Government of Niger regularly requests advice from this organization concerning women s issues. This case illustrates the sustainability of results and the spinoff from the NetCorps interventions i.e. the opening of a computer center, further training to the women of the association and an up-to-date web site, which had an impact on the organization s visibility in the country and the efficiency of their operations. Evaluation of the NetCorps Canada International Program (NCI) 14

27 Dorothy Gordon, Director General Advanced Information Technology Institute (AITI) of Ghana India Kofi Annan Centre Skills transfer and its importance Jumoke Ajayi, former Director of Women s Initiative for Self Empowerment (WISE) Volunteers must have adequate resources such as training equipment, tool kit, testers, etc. In Ghana, when donors are involved in ICT4D, they have to take into account the importance of skill transfers in their programs and determine the skill gaps in order to fulfill the local needs. At the same time, donors should be careful not to undercut the private sector with free professional technical support if the capacity exists at the local level. We need interns who can transfer intermediary skills and offer mentoring for larger ICT projects basic skills are available (locally) Encourage contacts between interns and host organizations prior to departure in order to clarify expectations Evaluation of the NetCorps Canada International Program (NCI) 15

28 4.4 Other Findings During the field visits, the following issues emerged, which would require attention if the program is funded in the future Selection of host organizations The sample of 31 organizations included four host organizations that are part of the UN umbrella, and therefore, already receive CIDA support through the multilateral channel. Based on our interviews with the representatives of these organizations, it is apparent that they have a significant budget for consultancy. In some cases, the NetCorps intern was hired as a consultant after his assignment. This outcome was in line with one of the objective of the NetCorps Program, i.e. increasing the intern s employability. We also noted that there were positive benefits for the organization. However, allocating an intern to this type of organization means that another recipient organization, which could not afford to pay for consultants, did not get ICT support from the program. Therefore, the criteria and processes used to select host organizations should be reviewed to ensure an appropriate match between the intern and host organization, and that the resources are used most effectively Pre-departure a) Adequacy of direct communication between volunteers and host organization During the field visits, the evaluation team found that some of the host organizations had difficulty formulating their ICT needs due to their lack of expertise. Many interns (88%) confirmed that there was a discrepancy between their job description and what they actually did overseas. The reasons identified by the interns for the discrepancies were the following: prerequisites were not in place in the host organizations (69.2 %); the host organizations priorities were not well defined (57.7%); the host organizations and interns expectations were not coherent with the organization s reality or actual needs (e.g. basic hardware) (34.6 %); and changes in organizational priorities (23.1%). Based on our analysis, the root cause for these discrepancies appeared to be the lack of direct communication between volunteers and host organizations prior to departure in order to clarify mutual expectations. Evaluation of the NetCorps Canada International Program (NCI) 16

29 b) Adequacy of volunteer profile, pedagogic training material and/or equipment Interviews with the host organizations and interns revealed that some of the volunteers didn t have pedagogic training and/or appropriate ICT material (such as diagram diagnostic tools, free software, testing equipment) before departure. In some instances, interns did not have the proper ICT experience and credentials to fulfill their mandate. 4.5 Benchmarking Benchmarking with ICT programs Benchmarking was used to compare the selection criteria of NetCorps and other ICT international volunteer programs. The performance of these different models was not compared since we would have to increase our sample in the field to a large extent including our budget. Table 9 shows that age and profile are the major differences between groups. GeekCorps, for example, recruits professional with at least 5 years experience in ICT, whereas NetCorps train students for a short period of time with postsecondary education and little experience in ICT. Table 9: Benchmarking with other ICT volunteer groups in Criterion Age Profile Length Engagement Approach NetCorps 19 to 30 Post-secondary graduate, comfortable with computers 3 to 6 months North-South GeekCorps IESC No age limit Professionals with at least 5 years experience in ICTs 4 to 9 months North-South South-South UNITeS No age limit Graduate students with ICT skills 4-24 months South-South approach prevailing Peace Corps No age limit No fixed criteria 2 years preceded by a 3 months incountry immersion North-South Trust of the Americas (previously NetCorps America) No age limit Volunteers from the South, usually graduate students Not available South-South Evaluation of the NetCorps Canada International Program (NCI) 17

30 4.5.2 Benchmarking with Youth Volunteer Programs The evaluation team also compared NetCorps with CIDA s International Youth Internship Program (IYIP). Like NetCorps (under Industry Canada), the IYIP is part of the Government of Canada s Youth Employment Strategy (YES), an inter-departmental program managed by Human Resources and Social Development Canada (HRSDC). The IYIP is an employment program for Canadians aged who holds a postsecondary degree. It offers young Canadians the opportunity to gain valuable international development work experience. Table 10 shows the many common characteristics between NetCorps and IYIP. Table 10: Comparison between NetCorps and IYIP in Criterion NetCorps IYIP (source CPB) Age 19 to to 30 Profile Post-secondary graduate Post-secondary graduate Length 3 to 6 months 6 to 9 months Approach North-South North-South Members organizations funded Budgeted Funding $17,000 per intern $15,000 per intern by CIDA Admin cost allowed 20% per intern 20% per intern by IC & CIDA Additional Cost $161,000 ( ) None allowed by IC related to cost Coalition Secretariat interns sent abroad 395 interns sent abroad * interns sent abroad 444 interns sent abroad *Estimate provided by Coalition just before the end of fiscal year Previous evaluations of NetCorps International Program Other sources of information, including two major studies, were used to verify and consolidate the findings from the present evaluation: the Formative Evaluation by Industry Canada 4 (2003) and the Internal Reflection of Coalition members 5 (2006). Many of the issues raised in these studies confirmed our findings and are still relevant. 4 Formative Evaluation of the NetCorps Canada International Program, Final version, Industry Canada, Audit and Evaluation Branch, April 30, Evaluation of the NetCorps Canada International Program (NCI) 18

31 4.6.1 Industry Canada s 2003 Formative Evaluation In 2003, Industry Canada required a formative evaluation of the NCI Program. Issues related to the Program s continuing relevance, design and delivery, program processes and preliminary impacts (e.g. impacts on the youth interns and overseas host agencies), and lessons learned were examined in the evaluation. a) Relevance The evaluation concluded that NetCorps was still relevant i.e. it was successfully implemented as intended from 1999 to 2003 and it was a creative approach for youth employment that did not duplicate other programs due to its focus on ICT. b) Program design and delivery Areas identified for improvements in the program design and delivery can be summarized as follows: 1. Ensure that all current NCI objectives are clearly defined; 2. Encourage the participation of more interns from all regions and client groups; 3. Refine the pre-departure orientation; 4. Provide more post-internship debriefing and assistance to interns, if feasible; 5. Increase the promotion of NetCorps in Canada; 6. Expand computer software options; 7. Clarify and strengthen the international development aspect of the NCI Program*; 8. Allow more flexibility in the duration of internships; 9. Examine the feasibility of increasing the NCI budget to reflect increases in travel and related expenses. *The evaluation team was not able to find evidence or follow-up of corrective measures taken following the formative evaluation. c) Program process and preliminary impacts The NetCorps internship appeared to help interns with their education and career development. NetCorps also had beneficial impacts on the host organizations. In particular, sustainable knowledge transfer occurred through the train-the-trainer component of the internship, for example, related to basic computer training and the maintenance of websites or databases. Respondents believed that NetCorps 5 NetCorps Canada International. An Internal Reflection to Maximize the Benefits of the Coalition, Heather Creech, IISD, March Evaluation of the NetCorps Canada International Program (NCI) 19

32 responded appropriately to the host organization s needs in the vast majority of cases Coalition members internal reflection 6 In an effort to determine the extent to which NetCorps had i) met its goals and lessons, ii) contributed more broadly to the sharing of knowledge on the role of volunteerism in the field of ICTs for development and iii) strengthened the case for future fundraising for NetCorps, the NetCorps Coalition members provided an internal reflection on NetCorps to the evaluation team. The main recommendations related to the program design and delivery can be summarized as follows: 1. Make adjustments to existing practices: Follow trends in ICT4D; Develop more specific TORs; Review the current reporting format to the funder with a view to moving it from a compliance approach to an outcome approach; Review secretariat. 2. Institute knowledge sharing and reporting: Develop e-learning events, e-conference among host organizations; Review existing ICT4D indicator initiatives for monitoring. 3. Do more together: Members to work together; Develop a strategic plan to engage with the ICT4D sector; Investigate foundations and other agencies for grants. This reflection demonstrates the Coalition members firm intention to work together to improve the NetCorps program. 4.7 Relevance and fit of NetCorps to CIDA s priorities Development priority The evaluation findings clearly demonstrate that NetCorps has had a beneficial impact on host organizations in terms of enabling results such as improvements in the efficiency and effectiveness of the management of host organisations. The ICT spin offs were clearly identified in the areas of accountability and transparency, which are link or contribute to the Good Governance of the host organizations. In particular, knowledge transfer has occurred through the train-the-trainer component of the internship, for example, related to basic computer training and the maintenance of websites or 6 NetCorps Canada International. An Internal Reflection to Maximize the Benefits of the Coalition, Heather Creech, IISD, March Evaluation of the NetCorps Canada International Program (NCI) 20

33 databases. Another example of sustainable outcomes can be found in spin offs which are unexpected benefits attributed to the presence of the intern. The creation of an ICT laboratory for youth and women in Niger is a clear demonstration of this principle ICT as a development tool At CIDA, ICT is considered a crucial tool to support development, which is in line with the Agency s medium and long terms priority sectors and programs i.e. to enhance transparency and accountability, capacity building and more particularly areas such as communication, productivity, accountability, capacity building and women empowerment. Host organizations satisfaction with the NetCorps internships demonstrates the niche for an ICT voluntary program. Currently, CIDA has no programs, per se, that address ICT as a means for development. Nevertheless, as part of the Canada Fund for Africa, CIDA has some specific contribution agreements with 4 partners, will end in 2008, namely: 1. Connectivity Africa (part of Canada Fund for Africa) 2. Enablis (part of Canada Fund for Africa) 3. epolicy Resource Network for Africa (epol-net: part of Canada Fund for Africa) 4. DOT (Digital Opportunity Trust) 4.8 Relation with CIDA s Volunteer Cooperation Program One of CIDA s Canadian Partnership Branch (CPB) responsibilities is to manage the Volunteer Cooperation Program (VCP). It has contribution agreements with over 256* agencies, including all individual NetCorps coalition members (*source CPB Branch). In 2006, the Volunteer Cooperation Agencies (VCA) (CWY, CCI, CUSO, OXFAM, VSO, CECI, CESO, SUCO, and WUSC) sent 2202 volunteers of all ages abroad for periods ranging from 3 to 24 months with a total contribution from CIDA of $41M. Five of the 9 VCA agencies are part of the NetCorps Coalition (CWY, CCI, CUSO, OXFAM, VSO). Also, the International Youth Internship Programs (IYIP) is part of the VCP program and funds some 118 organizations. Because IYIP is managed under CPB, where all NetCorps coalition members have contribution agreements, CIDA management could consider adding the NetCorps program to the IYIP. This would ensure a reduction in the management s level of effort. The following table 11 summarizes the financial value of contribution agreements for each member of the coalition and their corresponding CIDA delivery mechanism. Evaluation of the NetCorps Canada International Program (NCI) 21