EVALUATION FRAMEWORK MATRIX: EVALUATION QUESTIONS, RESPONDENTS, INDICATORS AND DATA SOURCES. Respondents. NCAs

Transcription

1 Data Source Indicators UNEP Finance Manager UNEP Task Manager NGOs Private Industry Min. Education? Steering Committee CONAP, OTECBIO, NCAs Team EVALUATION FRAMEWORK MATRIX: EVALUATION QUESTIONS, RESPONDENTS, INDICATORS AND DATA SOURCES s EVALUATION QUESTIONS A. Strategic Relevance 1. To what extent were project objectives and implementation strategies consistent with national and sub regional environmental issues and needs? 2. To what extent were project objectives and implementation strategies consistent with (i) UNEP s mandate and policies at the time; and (ii) the GEF Biodiversity focal area, strategic priorities and operational programme(s). 3. Were project objectives realistic, given the time and budget allocated to the project, the baseline situation and institutional context? 4. To what extent did UNEP and GEF engage NCAs and other national biosafety stakeholders in project design, implementation, monitoring and reporting? B. Achievement of Outputs 5. How successful was the project in achieving its planned outputs, considering aspects such as quantity, quality, sequencing, timeliness and usefulness? To what extent have project outputs contributed towards the expected outcomes? level of achievement of objectives and outcomes project design project delivery and level of achievement project delivery and level of achievement project delivery and level of achievement document, MTE, document, MTE MTE, PIRs and PDF C. Effectiveness: Attainment of Objectives and Expected Outcomes 6. How and to what extent did the project succeed in

2 (MAG) developing and implementing a framework for biosafety? To what extent has this strengthened the involvement of the NCAs in the implementation of the Cartagena Protocol on Biosafety (CPB)? 7. To what extent has the project had an impact on the development of capacities for biosafety management? 8. How and to what extent did the project build administrative capacities to handle requests, make informed decisions and communicate them to applicants and the BCH? 9. To what extent has the project ensured that decisions on LMOs are based on risk assessments, are timely, transparent and coordinated, and avoid duplicity or unnecessary bureaucracy? 10. To what extent did the project increase the capacity to monitor and ensure regulatory compliance? 11. Are sufficient technical and human capacities being put in place for risk assessment and management for decisionmaking, considering both traditional and new generation LMOs? 12. Are trans boundary movements of LMOs occurring in accordance with the CPB, and in a manner that is understood and accepted by the private sector (exporters /importers)? 13. To what extent have BS public sensitization and educational strategies been contributed to increase public awareness? To what extent is this leading to a change in human behavior? continued involvement of Public Health Min. and other NCAs in NBF # and outcomes of cases of liability/redress # of applications, communication of decisions # of applications, decisions and processing time trends in enforcement and compliance CAN agreements w/ CTNBio, TORs and # personnel involved number of documented trans boundary movements, enforcement of regulations for trans boundary movement of LMOs Results from any surveys conducted by project on public awareness and attitudes and changes to baseline situation document,, CONAP /BCH data, CONAP /BCH data CONAP and BCH data, PIR and s signed agreements w/ NCAs, personnel assigned. # trans boundary LMO movements. data from Min. of Education or other implementers 14. To what extent has the project contributed to increase ;

3 information sharing through greater access to biosafety information? D. Sustainability 15. Socio-political: Are there any social or political factors that influence positively or negatively the sustenance of project results and impacts? 16. To what extent did UNEP and GEF engage the participation of national biosafety stakeholders in project design, implementation, monitoring and reporting? 17. Is there sufficient government/stakeholder commitment to enforce and implement the programmes, plans, agreements, monitoring systems etc. prepared and agreed upon under the project? 18. Financial: To what extent is the continuity of project results and their impact dependent on continued financial support? Will adequate financial resources be made available to ensure the continuity of programmes, plans, agreements, monitoring systems etc. that were prepared and agreed upon under the project? 19. Institutional: To what extent is the sustenance of the results and progress towards impact dependent on national institutional frameworks and governance? To what extent are institutional governance structures and capacities in place to sustain processes, policies, agreements and legal/regulatory aspects that were supported by the project? Catalytic Role & Replication: Has the project had a catalytic role in promoting institutional change, changes in behavior, policy changes, new opportunities or follow-up support? E. Efficiency 21. Did the project apply any time or cost-saving mechanisms in order to achieve results within the approved timeframe and budget? institutional agreements; access and hits to online biosafety web page. continuity of project-supported initiatives workshops and consultation events during design phase policies of new government, budget and staff allocations expenditure and delivery trends, project work plans and budget revisions Final PDF Interviews project unit documentation, signed budget revisions, MTE, PIRs

4 22. Did the project face any obstacles (financial, administrative, managerial) and to what extent has this affected its efficiency? 23. To what extent did the delay in implementation affect the delivery of the project outcomes? 24. To what extent did the project succeed in securing the necessary funds to implement the educational strategy? 25. Were the required progress and financial prepared satisfactorily and submitted on schedule? F. Factors affecting Performance Preparation and Readiness: 26. Were the project s objectives and components clear, practicable and feasible within its timeframe? 27. What factors influenced the quality-at-entry of the project design, choice of partners, allocation of financial resources etc.? 28. Were the partnership arrangements properly identified and the roles and responsibilities negotiated prior to project implementation? Were counterpart resources (funding, staff, and facilities) and enabling legislation assured? Were adequate project management arrangements in place? Implementation and Management: 29. To what extent were the project implementation mechanisms outlined in the project document effective in delivering project outputs and outcomes? Were project expenditure and delivery trends, recruitment and procurement timelines project delivery trends (recruitment, procurement, contracts) in comparison with planned timelines Co-financing is made available. s submitted on time and accepted. project performance and delivery trends, positive appraisal of project document institutional arrangements and counterpart contributions clearly spelt out in project project performance and level MTE, PIRs financial. PIRs, financial project document, Quality Assurance assessment, MTE, PDF project document, PDF PIRs, MTE, Final

5 adaptations made to the approaches originally proposed 30. How effective and efficient was project management by CTNBIO and the PMU, and how well did they adapt to changes during the project lifetime? 31. To what extent did the Steering Committee provide guidance and contribute to effective project implementation? 32. To what extent did the project management and national partners respond to the guidance/recommendations provided by the Steering Committee, the UNEP Task Manager and Mid-Term Review. 33. Identify any operational and political / institutional problems and constraints that influenced implementation, and how the project partners tried to overcome these problems. Stakeholder Participation and Public Awareness 34. What approaches were used to identify and engage stakeholders in project design and implementation? 35. To what extent have project partners and stakeholders collaborated/interacted effectively during project design and implementation? 36. Did the project promote mechanisms for stakeholder participation in decision-making in the programmes, plans and other initiatives that it generated? Country Ownership and Driven-ess 37. To what degree has CONAP assumed responsibility for the project and provided adequate support to project execution, including the cooperation received from the of achievement of outputs/outcomes. implementation of SC decisions/recommenda tions implementation of SC/UNEP/MTE recommendations by PMU/CNAs perceptions; identified obstacles/constraints and remedial actions taken evidence of workshops or other consultation mechanisms documented interactions evidence of stakeholder participation in planning and decisionmaking Perceptions, performance of CONAP minutes of SC meetings minutes of NCC meetings, PIRs, MTE minutes of SC meetings, Pairs, MTE, Final PDF, PIRs, MTE PIRs, MTE, Final

6 various public institutions involved and timeliness of counter-part funding? 38. To what extent have the national and regional political/institutional frameworks facilitated project performance? Financial Planning & Management 39. Were sufficient financial resources made available and disbursed in a timely manner to the project and its partners? 40. Were administrative processes such as staff recruitment, procurement of goods and services (including consultants), and preparation/ negotiation of cooperation agreements conducted efficiently and in a timely manner? 41. Were co-financing commitments met as programmed and made available in a timely manner? 42. Were additional resources financial, in-kind leveraged by the project, beyond those that were already committed prior to the project s approval? 43. Identify irregularities (if any) in procurement, use of financial resources and human resource management, and the measures taken by CONAP or UNEP to correct/prevent such irregularities. UNEP supervision and backstopping: 44. Assess the quality and efficiency of UNEP s supervision plans, outcome monitoring, PIR reporting and financial/administrative services and project team in implementation, timeliness of project delivery consistency of NBFs in Central America, synergies with other countries through regional WB/GEF biosafety project timeliness of disbursements, budget revisions Budget revisions, increased allocations to existing/new budget lines through cofinancing Documented irregularities, interrupted procurement/disburse ment processes timeliness and acceptance of PIR and financial ; timeliness of disbursements and administrative support PIRs, budget revisions, financial PIRs, MTE, audit PIRs, MTE

7 Monitoring and evaluation> 45. Did the project s design include a viable M&E plan that is based on outcomes and includes indicators? services by UNEP Monitoring Plan is included in the project 46. Did the project s design include a monitoring budget? document includes monitoring budget line. 47. Have monitoring findings influenced adaptive management and contributed towards resolving implementation problems? 48. Are there specific indicators for each of the project objectives? Are the indicators measurable, attainable (realistic) and relevant to the objectives? Are the indicators time- bound? 49. Have the responsibilities for M&E activities been clearly defined? Were the data sources and data collection instruments appropriate? Was the frequency of various monitoring activities specified and adequate? In how far were project users involved in monitoring? evidence of technical/management decisions based on monitoring findings Indicators are included in Results Framework for each objective. Designated parties conduct monitoring activities periodically with inputs from project participants. The monitoring approach is considered methodologically appropriate by the evaluator and most respondents. document monitoring monitoring.