UNICEF Evaluation Office Evaluation of UNICEF s Policy Advocacy Work: Mapping and Scoping Exercise

Size: px
Start display at page:

Download "UNICEF Evaluation Office Evaluation of UNICEF s Policy Advocacy Work: Mapping and Scoping Exercise"

Transcription

1 UNICEF Evaluation Office Evaluation of UNICEF s Policy Advocacy Work: Mapping and Scoping Exercise The Evaluation Office is planning to evaluate UNICEF s policy advocacy work in Middle Income Countries (MICs). The purpose of this note is to outline a plan to: (i) map the policy advocacy efforts by UNICEF, (ii) distil, analyse, define and prioritise substantive issues and (iii) develop a Term of Reference for the planned evaluation. Policy Advocacy The Mission Statement of UNICEF states that UNICEF is mandated by the United Nations General Assembly to advocate for the protection of children s rights, to help meet their basic needs and to expand their opportunities to reach their full potential. 1 UNICEF Advocacy Toolkit defines advocacy as a deliberate process, based on demonstrated evidence, to directly and indirectly influence decision makers, stakeholders and relevant audiences to support and implement actions that contribute to the fulfilment of children s and women s rights. Advocacy is a means of seeking change in governance, attitudes, power, social relations and institutional functions. 2 In its Core Commitments for Children in Humanitarian Action, UNICEF defines advocacy as deliberate efforts, based on demonstrated evidence, aimed at persuading decision makers to adopt policies and take actions to promote and protect the rights of children and women in humanitarian situations. 3 The UNICEF Medium-Term Strategic Plan (MTSP) positioned advocacy as a standalone focus area. Focus Area 5 of the MTSP says that policy, advocacy and partnerships for children s rights is designed to strengthen the capacities of states and societies to design and implement social and economic policies, legislative measures and budgetary allocations that enable them to meet their obligations to children under the Convention on the Rights of the Child, the Convention on the Elimination of All Forms of Discrimination against Women and the Convention on the Rights of Persons with Disabilities. 4 A thematic discussion on results and lessons learned of focus area 5 explains that it was designed to strengthen the capacity of governments, agencies and societies to realize children s rights 5 and unpacks the focus area into four key components: 1. Data collection, analysis and dissemination, and child poverty and disparities analysis; 1 UNICEF s Mission Statement Mission%20Statement%20Poster.pdf 2 Advocacy Toolkit, A guide to influence decisions that improve children s lives, 2010, UNICEF 3 UNICEF Core Commitments for Children in Humanitarian Action. 4 Policy Advocacy and Partnerships for Children s Rights, Thematic Report 2012, 5 Thematic discussion on results and lessons learned in the medium-term strategic plan focus area 5: Policy advocacy and partnerships for children s rights, Background Document, E/ICEF/2012/CRP April 2012, Executive Board, Annual session 2012, 5-8 June

2 2. Research, analysis and technical advice on economic and social policy and issues; 3. Advocacy to inform national policy and expand fiscal space for children; 4. Fostering civic participation by children and young people. The UNICEF Strategic Plan, notes that the fundamental mission of UNICEF is to promote the rights of every child, everywhere, in everything the organization does in programmes, in advocacy and in operations. As per the UNICEF policy advocacy page on the UNICEF intranet, the objective of policy advocacy is to influence macro level policies that guide national frameworks, legislative reform and budgetary allocations affecting children and women, and advocate for actions, build capacities and leverage resources such that national and global investments contribute to fulfilling the rights of all children, including in emergency situations. 6 UNICEF has worked in recent years to support its work on advocacy, notably through a Toolkit. The organization is also currently in the process of developing its first-ever global advocacy strategy in support of the priorities of UNICEF s Strategic Plan. Middle Income Countries (MICs) and UNICEF The World Bank defines the world s Middle Income Countries (MICs), as having a per capita gross national income of US$1,026 to $12,475 (2011) which are a diverse group by size, population, and income level. Middle income countries are home to five of the world s seven billion people and 73 percent of the world s poor people. At the same time, middle income countries represent about one third of global GDP and are major engines of global growth 7. UNICEF s publication Policy Matters indicates that middle income countries represent one of the most challenging contexts to achieve equity for children. Income and social inequities in MICs are greater than in low- or high-income nations. Tackling inequities in MICs, which are home to almost 70 per cent of the world s poor, necessarily implies addressing some conceptual, operational and political challenges. MICs are often associated with emerging economies with rapid and sustained economic growth. Strong growth, however, does not imply that all children in the country are faring well. 8 In light of a rapidly changing world and the increasing number of MICs, the traditional role of UNICEF support to government is gradually shifting from Service Delivery to supporting national governments to meet their obligations under the Convention on the Rights of the Child. Advocacy to influence policy is now increasingly becoming a critical core of UNICEF essential business. Country Offices progress reports claims to engage in advocacy and the large majority of Middle Income Countries have it as their essential business as per their Country Programme Documents. Rationale In light of the increased shift from Service Delivery to Advocacy in MICs and in view of significant investments and claims about advocacy impacts in its reports and statements, the organization does not seem to have a well-defined frame of intervention for policy advocacy. Moreover, most claims 6 accessed 16 September (page 7) 2

3 have never been independently evaluated nor has there been an effort to objectively document lessons. By putting the focus of the evaluation on the role of advocacy in MICs will help UNICEF to objectively assess what has worked and what lessons can be learned in view of strengthening policy advocacy efforts in MICs, thereby improving the lives of children across the world. A comprehensive evaluation of UNICEF s policy advocacy work is complex and challenging and will require careful design and planning. The evaluation has to overcome four challenges prior to framing the scope and methodology of the evaluation: a. UNICEF has never fully documented the range of policy advocacy efforts it supports, nor articulated a corporate strategy for policy advocacy. b. While UNICEF offices make significant investments and claims about advocacy impacts in its reports and statements, most claims have never been independently evaluated nor has there been an effort to objectively document lessons. Therefore, there is little evaluative evidence to build upon for this comprehensive global evaluation. c. The architecture of development cooperation is complex in MICs, with countries at different stages of economic and social development. Furthermore, the context in which the country policy, and regulatory environment for children s wellbeing are diverse across the MICs. The diversity of the country context makes framing the evaluation scope difficult. d. Policy change are often complex processes across the sectors of the Strategic Plan of UNICEF, therefore, it is difficult to isolate the impact of interventions from the influence of other factors and actors making the subject matter very complex to evaluate. Given the above challenges the evaluation of UNICEF s policy advocacy should be seen as both as evaluating a complicated (having multiple components) and a complex intervention (as the issues are dynamic and emergent). Therefore, the EO proposes to undertake a mapping and scoping exercise in preparation for the upcoming evaluation. The assumption behind conducting this preliminary exercise is that knowing the range and distribution of the work being carried out by UNICEF in the policy advocacy area is essential to consider: The scope the proposed evaluation; Consider the strengths and limitations of potential evaluation methods and their best fit with the requirements of the evaluation being designed. Purpose The goals of the mapping and scoping exercise are twofold: 1. Support the development of a well-justified evaluation focus and approach through: A body of evidence (a map ) setting out the range and distribution of types of policy advocacy interventions pursued by UNICEF to inform the choice of types/locations/etc. of interventions to be evaluated. Distilling and analysing issues, clarifying concepts and definitions of policy advocacy (i.e. clarifying the distinction between policy and public advocacy and how they are applied in a variety of contexts). 3

4 2. Prepare well-focused and well-evidenced Terms of Reference (ToR) based on strategic and well-justified priorities for a comprehensive evaluation of UNICEF s Policy Advocacy work in MICs. Outline, the evaluation rationale, purpose and objectives, scope, methodology and timeframe of the evaluation after consultation with stakeholders; Establish a clear and concise evaluative framework to support the evaluation; Provide guidance on whether any specific issues should be addressed subsequently by supplementary evaluations; Provide guidance on approaches to promote the timely dissemination and utilization of the evaluation findings, lessons and recommendations; and Outline principles to support the meaningful participation of internal and external stakeholders in the evaluation, while respecting their heavy and demanding workloads. Methodology The mapping and scoping exercise will take a participatory approach. Desk Review: Undertake a preliminary desk review of data sources relevant to policy advocacy and provide initial analysis to support and guide the evaluation data collection mission. A method of analysis for documents will also be developed by the consultant undertaking the exercise. Semi-structured interviews: Discussions will be conducted (in person and over skype/phone) with UNICEF staff and partners at country, regional and headquarters levels. Particularly regarding priority issues for evaluation, information required (and in what format) and the evaluation scope, approach and process. A list of all the people to be interviewed at the HQ, RO and CO. Key people to be interviewed at HQ include the policy advocacy unit, social policy advisors, sector advisors, Department of Communication, Humanitarian colleagues. In the regions deputy regional directors will be targeted. At country level two or three countries (for skype/phone interviews) per region will be selected purposively on advice from the regional office for the mapping and the programmes officers and representatives will be contacted. The country selection is done based on the advice of Regional office in the assumption that RO knows about the activities of the CO. Interviews also serve as a mechanism to identify and select documents. Survey: A survey will be administered to obtain information from a larger sample of people. The selection will be based on key parameters (country size, economic status, status of children, type of governance etc.). A sampling methodology will be developed to that effect. Regional Visits: The consultant will visit three regional offices to more fully understand the range of activities. A mapping can only be developed from a visit to three regional offices as a smaller sample would be highly skewed. Visits also serve as a mechanism to identify and select documents. The regional office is selected on the basis of agreement and availability of RO staff. Workshops: Two workshops will be organized as part of the mapping, scoping exercise: 1. Half day workshops with programme staff handling various sectors will be organised at the RO. 2. Validation workshop with the evaluation reference group to vet the mapping exercise and the proposed evaluation approached outlined in the draft ToR. 4

5 Document repository: A document repository will be created. Coverage: Although the evaluation will focus on MICs, the mapping exercise will cover policy advocacy interventions across UNICEF interventions and geographic and socio-economic settings. Key Questions While a comprehensive mapping exercise will be undertaken to identify and better understand what UNICEF is doing in the area of policy advocacy, part of the exercise is also to maximise the use of the evaluation. Therefore, two different sets of questions will be posed during this exercise. 1. Proposed questions determining the utility of the exercise 9 Why is this evaluation important, and why is it important in 2015 specifically? (Evaluation Rationale) What will the evaluation seek to achieve, and to what end? (Evaluation Objective/Purpose) What will the evaluation cover and what will it not cover and why? Through what overarching questions will it seek to explore these areas? (Evaluation Scope) What data sources should be used, and in what ways, to answer the key evaluation questions? (Methods) What is the best timing in order to avoid undue burden on UNICEF staff during critical periods, as well as overlap with other major initiatives, and make the greatest potential contribution to decision-making and/or promote positive change within UNICEF? (Timeframe) What stakeholders should be involved in the evaluation in order to promote technical accuracy, relevance to the organization, and feasibility of action on its eventual recommendations? (reference group) Particular attention will be paid to the OECD/DAC evaluation criteria. Mapping exercise questions The key sub-questions, data sources, methods and type of analysis are illustrated in the matrix below. Mapping Matrix. 1: Have the policy advocacy objectives been defined and has the strategy to achieve them been conceptualized? Data Sources Methodology Analysis UNICEF documents Desk Review. Listing of related to MICS and Semi-structured objectives/strategies policy advocacy at interviews. stated in documents at HQ, RO, and Survey/questionnaire. various levels. selected CO Web site Key informant interviews. 9 The scoping exercise will further refine the set of proposed questions. 5

6 2: What are the substantive issues, methods, activities and process that have been put in place for policy advocacy? Stakeholder engagement at Regional and Country level on the range of activities at CO. CO plans, project documents, progress reports, country studies. Discussion with the Regional Office staff. Desk review; and follow-up telephone interviews with senior country office staff and selected government officials. Survey/questionnaire Listing of key methods, activities and process, in relation to key contextual factors. Analyze areas of convergence and divergence in issues, methods and process. 3: Who are the key partners and stakeholders? UNICEF documents and progress reports. Discussion with the Regional Office staff. Desk review; and follow-up telephone interviews with senior country office staff. Key actions done with UNICEF partners. Compile a list of key stakeholders and the type of collaboration. The mapping exercise should also raise the following questions (this list is not exhaustive and should be further refined during the scoping): 1. What problems or challenges did policy advocacy work intend to address? How were these problems/priorities identified? 2. Which stakeholders (including government institutions, civil society and private sector bodies, UN agencies, donors, foundations, global funds) participated in the policy advocacy efforts? Why were they involved and what were the gains from their involvement? What was their role? How were they involved, ways and capacities in which they participated? At what stage of the process were they involved? 3. What interventions and/or activities were implemented? 4. What approaches, methods and/ or strategies were used to implement interventions? 5. What was the duration of advocacy interventions (long versus short term)? 6. What was the geographical and sectoral coverage of advocacy interventions? 7. What was the development context (stable, conflict affected, humanitarian, MICs etc.) 8. What efforts were made to integrate equity, gender and child rights into the policy advocacy? 9. What documents exist in your office or elsewhere that reflect the policy advocacy with regard to the efforts you mentioned? 6

7 Outputs The mapping and scoping exercise will produce: 1. An Inception Report: A brief report outlining the consultants understanding of the work, proposed methodology, detailed schedule, itinerary and logistics. The report will also include the following mapping and scoping tools (this list is not exhaustive): Sampling methodology (for field visits and survey respondents), Templates to guide the preparation and analysis; Interview guides; Design questionnaire/survey; Systematic analysis of documentation using templates; Preparation of mapping/matrices collating, analysing and synthesising data collected. 2. A policy advocacy map or set of matrices indicating the range and distribution of types of policy advocacy interventions in UNICEF. This output will be used to frame the substantive issues and scope, select the methodology, and develop a plan for the evaluation. The map will inform the choice of types and locations of interventions to be evaluated, and support the development of a well-justified evaluation focus and approach. These characteristics will be better understood after the mapping exercise as issues and methods of policy advocacy becomes clearer. The map or set of matrices will provide clear, well categorised quantitative and qualitative data on the following: Policy advocacy objectives and targets (strategic intent) [the why what needs to be changed]; Policy advocacy interlocutors [the who the individual or institutional targets to be influenced]; Activities and interventions [the what what we actually do]; Approaches and methods [the how the rationale and methodology guiding actions. E.g. Strategic or opportunistic? Transactional or positional? Based on research or evidence? Or idiosyncratic/based on personalities? Etc.]; Time frames and duration [the when long haul or short. Link to e.g. electoral cycles]; Geographical and sectoral distribution [the where]; Development context: e.g. stable, humanitarian, conflict affected etc. [ i.e. the where again]; Linkages: in methods (e.g. combined with public advocacy/campaigning); in partnerships; in locations (local/national/regional/global) etc.; 3. A preliminary desk review of available data and information to support the evaluation data collection mission, including the creation of a document repository; 4. A short analytical paper based on mapping, preliminary desk review and regional visits distilling and clarifying the concept, application and use of policy advocacy in UNICEF (clarifying the relationship between policy advocacy versus public advocacy). This paper will also summarize activities undertaken, key conclusions that cover specific objectives above and any recommendations not covered. 7

8 5. Well-evidenced draft Terms of Reference (ToR) for the evaluation that clarify the purpose of the evaluation and the main evaluation questions to be explored. The draft ToR will propose and appropriate design, including a coherent plan and organizing structure for the evaluation, coupled with a clear evaluative framework (Theory of change or other) and a preliminary mapping of key stakeholders. In short the TOR will reflect the evaluation utility, scope, appropriate methods, timeline and expertise required. Consultant Team Composition and Required Competencies The mapping and scoping exercise will be carried out by a team consisting of the following: One senior-level international Team Leader (39 Days) with the overall responsibility for providing guidance and leadership and for coordinating the analysis and preparation of the mapping and Terms of Reference. The Team Leader will be responsible for undertaking the mapping and scoping exercise from start to finish, for the bulk of data collection and analysis, as well as report drafting in English. The Team leader will be responsible for delivering all the outputs outlined in this ToR. The Team Leader will also oversee and manage the work of the data analyst. The team leader will have the following qualifications: - A strong team leadership and management track record (at least 15 years experience) in policy advocacy; - Solid understanding of development and child rights as a practice area; - Evaluation expertise with strong mixed-methods evaluation skills and flexibility in using non-traditional and innovative evaluation methods; - A strong commitment to delivering timely and high-quality results. - Extensive technical and practical development expertise, and familiarity with UNICEF s country-level operations; - In-depth knowledge of the UN s human rights, gender equality, equity and humanitarian agendas; - Commitment and willingness to work in challenging environments and independently, with limited regular supervision; - Good communication, advocacy and people skills; ability to communicate with various stakeholders and to express concisely and clearly ideas and concepts in written and oral form; - Language proficiency: Fluency in English is mandatory; good command of French and/or Spanish is desirable. One Data Analyst (24 Days) with at least 5 years experience required to undertake a preliminary desk review of available data and information in consultation with the selected Regional Offices and relevant Country Offices as well as in Headquarters, while supporting the senior consultant in preparing the mapping and the Terms of Reference. In close cooperation, and under the supervision of the senior consultant, the data analyst is expected to be responsible for: 8

9 - Undertake preliminary desk-review of data sources relevant to the evaluation and provide supplementary analysis to support and guide the evaluation data collection mission. - Support the preparation of the Mapping exercise. Prepare mapping matrices, analyze and synthesize data. - Support the preparation of Terms of Reference. The data analyst is required to conduct the data collection and assemble relevant data and information. Timeframe and Management The mapping and scoping exercise will be managed by the Evaluation Office, to be managed by the Senior Evaluation Specialist (Institutional Effectiveness). The exercise will be conducted by a Senior Consultant and Data Analyst from April 13 to June 29. The following steps are envisaged: Activity/Deliverable Deadline Senior Data Analyst LOE Consultant LOE Finalize Mapping and Scoping ToR and 2 April recruit consultant Initial briefing for specification of expected output; 13 April 1 day 1 day Collection of documentation, phone interviews with HQ and Regional locations An Inception Report including the mapping and scoping tools: Develop sampling methodology, Develop templates to guide the preparation and analysis; Develop interview guides; Design and administration of questionnaire survey; Systematic analysis of documentation using templates; Preparation of mapping/matrices collating, analysing and synthesising data collected Review of Inception Report (EO) 20 April 5 days 3 days 27 April 4 days 4 days 1 May 9

10 Visit to HQ Data Analyst only 05 May to 07 May 3 days (no regional visits): Visits to three Regional Offices in addition to phone interviews with the remaining Regional Offices and a select number of country offices. 05 May to 25 May 15 days Submission of deliverables: Desk Review of available data and information; Draft Analytical Paper; Draft ToR; 08 June 10 9 days: continuing data review and analysis and prepare mapping matrices, analyse and synthesise data Review of draft deliverables 12 June Debriefing 19 June 2 2 Final ToR 29 June 2 2 Total number of days 39 days 24 days Accountabilities The Senior Consultant shall work under the direct supervision of the Senior Evaluation Specialist for the Institutional Effectiveness Portfolio. Application Process All interested and eligible candidates should send an application including the following: a. Updated CV/Resume, and completed Personal History Profile (P11 ), b. A sample report of a similar exercise/subject or an evaluation report, with a clear indication of the applicant s contribution in the report. c. Availability and fee. The application should be transmitted via to by 2nd April, 2015, at the very latest, using the following evalofficeapplications@unicef.org Candidates should also indicate in the subject the consultancy they are applying for: Subject: Application for Policy Advocacy Scoping Senior Team Leader Or Application for Policy Advocacy Scoping Data Analyst 10