Annual Report of Evaluations

Transcription

1 Annual Report of s P age

2 Contents Contents... 2 Foreword... 3 Introduction... 4 ESCWA s Capacity... 4 Functional Entity... 4 Budget... 4 Policy Professionalization of the Function... 5 Roster... 5 Best Practices Portal... 5 Training Monitoring &... 6 Participation in UNEG meetings, surveys and teams... 6 Discretionary Self in ESCWA... 6 of ESCWA s Inter Governmental Mechanism... 7 Selection of Key Findings... 7 Other s... 8 Development Account (DA)... 8 OIOS Report Strengthening the Role of and the Application of Findings... 9 Institutional Findings & Conclusions Annex 1 Activities Annex 2 Workplan for P age

3 Foreword ESCWA s inaugural Annual Report of s marks an important juncture in ESCWA s strategic approach to evaluation. It follows the extensive review of the evaluation function undertaken in 2010, in which Accountability, Credibility and Continuous Learning were identified as the pillars of ESCWA s Policy. This report highlights the evaluation activities undertaken by ESCWA in 2010, and presents key findings that have institutional-wide relevance for the organization. Two leading activities shape this inaugural report. First, ESCWA s Policy was extensively revised following consultation with a range of internal and external partners. Here I would like to express my appreciation to UNEG colleagues, OIOS, ESCWA s member countries, and our fellow Regional Commissions, for their frank and constructive feedback and support in this endeavour. Second, member countries requested ESCWA to evaluate the effectiveness of its intergovernmental bodies. This proved an invaluable opportunity to enhance engagement with member countries on monitoring and evaluating the organisation s programme performance. It is therefore fitting that this activity is the key evaluation highlighted in ESCWA s inaugural Annual Report of s. ESCWA is committed to enhancing its impact by delivering high quality, relevant, and tangible advice serving member countries as a centre of excellence on economic and social development priorities in our region While our new approach to evaluation takes a more holistic and thorough review of how well we do business, I attach great importance to moving towards measuring the results of that business on our member countries and the people of the region. This is our challenge for the next Biennium. We look forward to your continued cooperation and interest in the work of ESCWA. Rima Khalaf Under Secretary-General Executive Secretary, UN ESCWA 3 P age

4 Introduction This is the first Annual Report of s published by the UN Economic and Social Commission for Western Asia (UN ESCWA). As per the revised Policy issued in September 2010, ESCWA will produce an Annual Report to highlight key achievements on enhancing programme implementation. The report synthesizes findings and recommendations from the range of evaluation activities conducted during the year, and identifies conclusions relevant to improving institutional processes. The purpose of evaluation at ESCWA is to promote a culture of innovation and adaption to the changing environment in our region. s review how the work and resources of the organization contribute to improving economic and social development in ESCWA s member countries. ESCWA values the perspectives and contributions of its stakeholders in enhancing its performance, in particular its member countries who direct ESCWA s work programme. Given that evaluations are a key entry point for stakeholder ownership of ESCWA s activities, this Annual Report intends to shed light on progress and achievements to its stakeholders in an easily accessible form available on ESCWA s internet. ESCWA s Capacity Functional Entity The evaluation function in ESCWA sits within the Programme Planning and Coordination Section, of the Programme Planning and Technical Cooperation Division (PPTCD). In this capacity, evaluation is an integral component of the programme and monitoring function, thereby creating an enabling environment for the integration of lessons learned and best practices into the planning function. This reflects ESCWA s approach to programme planning with evaluation as the third pillar of the planning cycle. Additionally, this provides a link between the regular programme of work, and the programme of technical cooperation and extra-budgetary activities which are also managed by the division. The new evaluations approach adopted in 2010 was implemented without additional staff resources, and utilising existing staff capacity. However a recommendation has been made to enhance staff resources to ensure that the evaluation function is adequately resourced given the broad scope of the evaluations programme. All regular programme, technical cooperation, development account and other extra-budgetary evaluations will be managed by PPTCD. PPTCD currently has 1 P4 (50%) and 1 P2 (50%) assigned to evaluations. Budget ESCWA s evaluation budget (excluding staff costs) for 2010 was $41,000. In order to address shortfalls in the current budget for the evaluations programme, ESCWA has devised an interim cost-sharing exercise with the substantive programmes to implement the revised evaluation approach in P age

5 Policy 2010 As an ongoing process of review to align to the UNEG Norms and Standards for, and in recognition of developments in UN reform, ESCWA s Policy was updated and approved in September Additionally, a guide to evaluations for programme staff was developed as an annex to the policy to assist staff to implement and follow up recommendations made through review of subprogramme activities at the broadest level, and to effectively manage and use evaluation findings in their work. The purpose of evaluation in ESCWA is to promote a culture of innovation and adaptation to the changing environment of the region. The new policy is built around three core principles: enhancing accountability, credibility and continuous learning. The objective of ESCWA s Policy is to introduce a systematic framework to enhance ESCWA s reputation as an organization committed to continuous learning and improvement. Central to this philosophy is increasing the independence of ESCWA s evaluative processes, increasing ESCWA s accountability to its member countries, and clarifying the lines of responsibility for management response to evaluation recommendations. As envisaged with the interim policy published in August 2008, ESCWA will regularly review the extent to which the policy reflects internal work practices. Continuous efforts to improve ESCWA s evaluation activities will be made, and resources needed to enhance the process for improving ESCWA s continuous performance will be dedicated for this purpose during the development of the Programme Budget. The policy was revised in conjunction with ongoing efforts to enhance ESCWA s programme performance and builds upon previous ESCWA policies, models of good practice in UN and other international entities. ESCWA s policy was revised with the kind assistance of many UNEG members, and from OIOS in particular. This support was highly appreciated. Professionalization of the Function Roster A generic profile for evaluation consultants was established which is established and managed by PPTCD on ESCWA s electronic roster system. ESCWA s evaluation roster has been developed in line with UNEG guidance on generic criteria for professional evaluation. ESCWA has advertised widely to UNEG members, professional evaluation associations, and UN vacancy distribution lists, as well as ESCWA s public website. By the end of 2010, fourteen consultants out of a total of sixty applicants to the roster were approved, as a result of extensive outreach to professional evaluation associations, UNEG, OIOS and other UN websites. It is important to highlight that the region does not have a professional evaluation association. Continued efforts in 2011 will broaden the reach of ESCWA s roster to other highly qualified candidates. Best Practices Portal A series of reports have concluded by recommending i that the regional commissions strengthen their websites as platforms for the exchange of regional best practices and lessons learned ii. The OIOS Inspection of the Programme and Administrative Management of ESCWA iii in 2006 outlined the requirement to develop lessons-learning guidelines that establish a framework for identifying, documenting and disseminating best practices and lessons learned, as well as using these lessons when planning and implementing programme operations and activities. iv ESCWA has been developing the framework for the best practices function during A concept note has been developed, resulting from the research of a wide range of UN and non-un entities websites and mechanisms. It is anticipated that the framework will be piloted by the end of the first quarter of P age

6 Training Monitoring & ESCWA engaged the services of expert consultants to conduct training on Monitoring and for staff across the organization. This will follow a two-day training on Project Design for the same audience. The modules of these training programmes were developed in the last quarter of the year, and will be delivered as a two day training January These modules were prepared in-line with the revised Policy, to provide participants mutually reinforcing resources which enhance their skills in project design, planning, implementation and evaluation. Participation in UNEG meetings, surveys and teams ESCWA participated in the Practice Exchange (EPE) and Annual General Meeting (AGM) of UNEG held in Vienna in May These meetings provided many valuable opportunities to liaise with evaluation entities and share lessons learned for the development of evaluation policies and the establishment of core professional evaluators. While ESCWA did not nominate staff to participate as members of a UNEG Taskforce, ESCWA may do so in 2011 subject to additional staff resources. In addition, ESCWA will deliver a presentation to the next UNEG EPE in March 2011 on the revised policy and approach for evaluation of normative work. Discretionary Self in ESCWA Prior to the articulation of the 2010 Policy, ESCWA s discretionary self evaluations were limited in scope and did not contribute to a learning process at the institutional level. The revised approach will consider: the efficiency of utilization of resources; the utility and reliability of the planning process for defining the programme of work; consultation with ESCWA s member countries in setting the programme s strategic direction; engagement with other regional actors; and identifying good practices for future programming. These evaluations are designed to complement the monitoring activities of the PPR, and occur between the two-year planning and reporting cycle. The first review period will be and will begin in These evaluations will also assess the effectiveness of the Regular Programme of Technical Cooperation (Section 22) in complimenting ESCWA s regular programme of work (Section 21). The results of these evaluations will derive institutionalwide learning which will be analysed by management, to ensure that future planning and programme management is enhanced throughout the organization. 6 P age

7 of ESCWA s Inter Governmental Mechanism ESCWA s Commission, comprised of its fourteen member countries v, is responsible for strategic guidance and oversight of ESCWA s programme of work. The intergovernmental body may request project, programme or other level evaluations by resolution, and specialized committees of the intergovernmental mechanism may refer suggestions to the Commission for consideration for evaluation. The Commission requested the ESCWA Secretariat to carry out an in-depth evaluation of the intergovernmental structure of ESCWA in the light of both programme priority, as identified by member countries, and ongoing United Nations reforms ; and decided to review the outcome of the in-depth evaluation at the 26th Session of the Commission. vi This evaluation was the first of its kind proposed for a Regional Commission, and the contribution of colleagues in ECLAC, ESCAP, ECE, ECA and the Regional Commissions Office in New York provided the vital context setting for considering the potential strengths and weaknesses of governing body structures at the regional level. 4. The Commission has a critical role to play in enhancing synergies and linkages between the work of LAS and its various specialized entities, with the work of the UN agencies, funds and programmes at the regional level. 5. Enhancing the agenda setting and discussions around outcome-oriented topics in line with setting the vision for programme direction 6. Systematisation of the protocol related functions of the IG bodies and subsequent training for the Commission participants to enhance knowledge of the working of UN bodies, including the development of a Policy Manual for the Commission The evaluator benefitted from observing the 26th Commission Session in May The final report is due in January 2011, and will be reviewed by ESCWA s Technical Committee for subsequent consideration by the Commission Session in May Selection of Key Findings 1. ESCWA IGM structure is well established, and mechanisms exist to review plans and to monitor and assess implementation of work. 2. Continued political insecurity in the region presents significant challenge in effectively addressing the development challenges of the region. This calls for creative approaches for both the structure and approach of ESCWA s management and presence. 3. Existing cooperation arrangements between ESCWA and other regional organizations, in particular the League of Arab States, will need to continuously evolve and mature to enhance the ability of member countries to utilize the various strengths of each partner. 7 P age

8 Other s Development Account (DA) ESCWA s Development Account guidelines were developed in 2010 in response to a need to enhance the global guidance provided by UN HQ for the preparation of project proposals. ESCWA is developing guidelines for sub-programmes to undertake an internal mid-term review of projects to take measures to address any constraints to implementation. One DA activity was evaluated in 2010 under the revised evaluation policy approach, with one other evaluated prior to the implementation of the new policy. From these evaluations, some selected conclusions and recommendations have been drawn with ESCWA-wide relevance: Monitoring systems need to be reviewed and enhanced for implementation of projects outside of ESCWA s base in Beirut. Noting that the region is subject to instabilities, building effective partnerships with other organizations with shared mandates may enhance monitoring and reporting. ESCWA has achieved a good balance of flexibility in its project development and workplans. Additional efforts will be made to identify and manage forseable risks to project implementation within the context of an institutional business continuity plan. Additionally, ESCWA project activities can improve by outlining a clear and timebound exit strategy of projects. There is a need to strengthen the professional and support capacity of ESCWA to plan, implement and monitor projects on a regular basis. Systemization of lessons learned is a priority for the application of findings to future activities ESCWA will review its DA Guidelines to enable more detailed and objectively verifiable logical frameworks of the project documents While not required by the DA Guidelines from HQ, ESCWA will, where possible, introduce a mid-term review of progress to be able to address any constraints to implementation with corrective measures in a timely manner. ESCWA also remains committed to supporting UN DESA in developing systems which help to quantify impact of DA-funded activities. ESCWA needs to identify intuitive and quantifiable tools to measure how its technical advisory and normative work can be attributed to influencing government policy in the region. As the needs of ESCWA s member countries are diverse (for example the differences between the need for ICT development in Kuwait versus Yemen), both the scope of activities and duration of projects must be developed in collaboration with target countries to ensure that the approach and outputs are realistically calibrated. Sustainability of activities post ESCWA s funding is dependent on identifying local level actors with the authority, coverage of responsibility, and commitment of national resources for the activity as part of an institutional workplan. It is critical to obtain this commitment from the outset of the project design stage. 8 P age

9 OIOS Report Strengthening the Role of and the Application of Findings OIOS is required to report to the General Assembly on a biennial basis, on the current state of evaluation in the UN Secretariat, provide a summary of key findings, conclusions and recommendations of evaluations completed in the biennium, assess the quality of said reports, and examine the application of evaluation to programme design, delivery and policy directives in the UN Secretariat. The key findings of this report identified that ESCWA s experience is one that is fairly consistent across the UN Secretariat: UN Secretariat entities focused on issues of programme implementation, outputs and outcomes and was guided by the criterion of effectiveness in assessing performance; For this exercise, ESCWA submitted four reports which were derived from the discretionary evaluations conducted in Of the 32 entities reviewed, ESCWA was one of five whose evaluations were of an internal nature only. Close to 75% of the reports reviewed addressed matters below the level of a programme or department. In the context of ESCWA s revised Policy issued in 2010, ESCWA will therefore be ahead of the curve in high level programme review. The report notes that ESCWA s allocations for evaluation have remained fairly consistent over the past three Biennia, with 0.8% in , 0.6% in , and 0.69% planned for Notably, according to the report, ESCWA is one of the thirteen programmes of the Secretariat where the staff working on evaluations had a range of other duties. ESCWA is also one of the fifteen programmes of the Secretariat which had an evaluation policy, and ahead of the more than half of the Secretariat programmes with a specific evaluation policy. OIOS has scheduled the next monitoring and evaluation inspection of ESCWA for While all Secretariat entities are engaged in evaluation activities, the quality of the reports varied widely across the respective entities; Resources for evaluation in Secretariat planned for declined compared with The proportion of the budget devoted to monitoring and evaluation activities resources decline from 0.42 ( ) to 0.36 ( ) percent. The results of evaluations contributed to building of evaluation capacity and quality, however evaluation has yet to become a fully accepted management function.. 9 P age

10 Institutional Findings & Conclusions The revision of ESCWA s Policy provided the organization with a practical tool with which to apply UNEG s Norms and Standards, as well as ESCWA s efforts to enhance accountability, credibility and continuous learning. Preliminary results - via the completion of one DA evaluation, as well as preparations for the broad approach to discretionary self evaluations - have provided a valuable opportunity to pilot the revised approach. Continuous review and adjustments will be made to support substantive work programmes on evaluations as requested. In addition, substantive functions tasked with the responsibility of designing and implementing activities have valued the additional support from the evaluation function, which highlighted that an internal gap existed for additional specialised support for the design and management of evaluation activities. Additional capacity building is required both inhouse as well as to member countries, for the identification of risks as well as mitigating measures to ensure that potential threats to project success can be anticipated and countered. ESCWA s revised evaluation policy approach requires significant dedicated resources to implement. In addition to identifying and managing evaluation expertise, policy development, and capacity building support to substantive programmes and member countries where requested, the current resources will need to be used as effectively as possible while building up the programme. ESCWA management will review the resources required to enhance ESCWA s programme implementation and recommend adjustments accordingly. For both regular and extra-budgetary activities, internal processes have highlighted that global instructions for programme design and implementation need to be further detailed for ESCWA s purposes. This has been done through the revision of the Development Account Guidelines (July 2010). This is particularly valid for measuring the impact of ESCWA s normative work programme. ESCWA will begin a process of contextualising global instructions as well as internal approaches, to operationalise and implement a results based management approach. 10 P age

11 Annex 1 Activities 2010 Title of Activity Subprogramme Status Category Regular Programme 1 Development of Methodology for Subprogramme s 2 of Subprogramme 5 - Statistics for Evidenced-Based Policy Making 3 Interim Programme Performance Report (PPR) 4 of ESCWA s Inter- Governmental Mechanism (IGM) Development Account Institutional In progress Discretionary Self Statistics for Evidenced- In progress Discretionary Self Based Policy Making Institutional Completed Mandatory Self Institutional (requested In progress Discretionary Self by ESCWA s Commission Session) 5 Strengthening of the development of International Merchandise Trade Statistics and the Compilation of e- commerce in Member Countries of ESCWA 6 Strengthening National Capacities in Environment Statistics, Indicators and Accounts in ESCWA and ECLAC and lessons learned Statistics for Evidenced- Based Policy Making Statistics for Evidenced- Based Policy Making January 2010 November 2010 Mandatory Project Mandatory Project 7 Knowledge Networks for Disadvantaged Communities Extra-budgetary (XB) Information & Communications Technology Division In progress Final Report Due January 2011 Mandatory Project n/a 11 P age

12 Annex 2 Workplan for 2011 Title of Activity Division Title Schedule Category Programme of Work Regular Budget & Technical Cooperation 1 Development of Methodology for Subprogramme s 2 of Subprogramme 5 Statistics for Evidenced-Based Policy Making 3 of Subprogramme 1 Integrated management of natural resources for sustainable development 4 of Subprogramme 3 Economic development & integration 5 of Subprogramme 6 Advancement of women 6 Programme Performance Report (PPR) for Biennium 7 of ESCWA s Inter- Governmental Mechanism (IGM) 8 of Programme Planning & Technical Cooperation Division (PPTCD) Development Account Institutional May 2011 Discretionary Self Statistics Division May 2011 Discretionary Self Sustainable Development & Productivity Division April 2011 Discretionary Self Economic Development & Globalisation Division June 2011 Discretionary Self ESCWA s Centre for August 2011 Discretionary Self Women Institutional December 2011 Mandatory Self Institutional (requested by In progress Discretionary Self ESCWA s Commission) n/a Pending budget Discretionary Self availability 9 Regional Harmonization of Cyber Legislation to Promote the Knowledge Society in the Arab World Information & Communications Technology Division November 2011 Mandatory Project 10 Participatory Human Development in Post Conflict Countries Extra-budgetary (XB) Social Development Division November 2011 Mandatory Project 11 of the Information Communication and Technology for Education in Iraq Project Emerging and Conflict Related Issues Section February 2011 Joint Project with UNESCO Funded through the Iraq Trust Fund (ITF) i Official Records of the General Assembly, Sixtieth Session, Supplement No. 16 (A/60/16, para. 139). ii Report of OIOS on the evaluation of linkages between headquarters and field activities: a review of best practices for poverty eradication in the framework of the United Nations Millennium Declaration (E/AC.51/2005/2, para. 75) iii A/64/63 Strengthening the role of evaluation and the application of evaluation findings on programme design, delivery and policy directives Report of the Office of International Oversight Services and A/61/61 Report of the Office of Internal Oversight Services on the inspection of the programme and administrative management of the Economic and Social Commission for Western Asia iv IED Report of the Office of Internal Oversight Services on the Thematic of "Lessons learned: Protocols and Practices" v Bahrain, Egypt, Iraq, Jordan, Kuwait, Lebanon, Oman, Palestine, Qatar, Saudia Arabia, Sudan, Syria, UAE, and Yemen. vi Pursuant to ESCWA resolution 291 (XXV) 2008 on the Frequency of the Sessions of the Commission and its Subsidiary Bodies 12 P age