2.1 Specific objectives of the evaluation The specific evaluation objectives are:

Size: px
Start display at page:

Download "2.1 Specific objectives of the evaluation The specific evaluation objectives are:"

Transcription

1 TERMS OF REFERENCE FOR A CONSULTANCY TO CONDUCT END OF PROJECT EVALUATION FOR THE AGRICULTURAL TECHNOLOGY AND AGRIBUSINESS ADVISORY SERVICES (ATAAS) PROJECT 1.0 Background and context The Agricultural Technology and Agribusiness Advisory Services (ATAAS) Project was designed to support the implementation of the national programs of the National Agricultural Research Organization (NARO) and the National Agricultural Advisory Services (NAADS). Arising from the restructuring of the NAADS program, the Government and the World Bank agreed to review the ATAAS project. The project restructuring led to the change of the implementing institutions from NAADS and NARO to MAAIF and NARO. The project is funded by a World Bank / International Development Association (IDA) loan and Global Environment Facility (GEF) grant. The project was developed as an investment to raise the level of productivity and income through the development and adoption of modern farming technologies and strengthening market linkages. The Project Development Objective (PDO) is to increase agricultural productivity and incomes of participating households by improving the performance of agricultural research and advisory service systems in the Republic of Uganda, while the Global Environmental Objective (GEO) is to enhance the environmental sustainability and resilience of agricultural production to land degradation and climate risks. The restructured project has four components: Component 1 is developing agricultural technologies and strengthening the NARS; Component 2 is enhancing partnerships between agricultural research and other stakeholders; Component 3 is strengthening agricultural support services; and component 4 is Program Management, Coordination and M&E. Component 1 is primarily the responsibility of NARO, while component 3 is the responsibility of MAAIF, and component 2 and 4 are jointly implemented by both institutions. 2.0 Purpose and Objectives of the Evaluation The project evaluation is targeted at establishing key achievements realized in implementation of the ATAAS Project. The purpose of the evaluation is to assess the overall project performance and to provide feedback for new strategic directions or implementation designs and delivery. The evaluation will assess what has worked, what did not work so well, key lessons learned and what should be emphasized or adjusted, in the second phase. This will essentially cover the analysis of the process of implementation, the changes that have occurred as a result of the project s intervention, opportunities and constraints that have been encountered, important lessons that have been learnt and recommendations for future design and implementation 2.1 Specific objectives of the evaluation The specific evaluation objectives are: 1

2 i. To assess the relevance and appropriateness of the approaches employed by ATAAS to achieve the project objectives and to assess how well the ATAAS addressed the key issues or problems identified at the inception of the project; Considering the theory of change employed in ATAAS project design and using program theory, establish the coherence and accuracy of the intervention logic (why the interventions were believed to cause the desired change) ii. To assess the effectiveness/extent to which the project objectives as spelt out in the project proposal were achieved; Critically assess the performance of the ATAAS Project in meeting its Development and Global Environment objectives based on the Key Performance Indicators as stipulated in the Results Framework; assess the action model of program theory -appraise the institutional and organizational arrangements/partnerships as well as the operational arrangements in the project implementation and administration that were critical in meeting the project objectives. iii. To review the efficiency in the utilization of resources to produce the required quality of outputs/outcomes; Assessing whether the project has been cost effective in achieving the desired outcomes; undertake operational efficiency and financial sustainability - an analysis of the financial performance and operational efficiency of the project including unit costs of service delivery as well as reviewing the PM&E system. iv. Critically assess whether the project outputs, are beginning to yield into outcomes, both intended and unintended; To identify and describe the outcomes of ATAAS on the targeted stakeholders along the value chain (farmers, traders, processors, among others); v. To assess ATAAS project sustainability; to assess the extent to which ATAAS s contribution on the targeted stakeholders will continue after the end of support so as to gauge the sustainability of ATAAS s work in the project area. vi. To evaluate the extent to which the ATAAS has successfully mainstreamed gender analysis and gender sensitive programming, and the extent to which it has successfully advanced gender equity. vii. To document key challenges, best practices and lessons learned and make recommendations that may be helpful in the design of the next phase of the ATAAS Project or other future projects. 3.0 Scope and focus of the Evaluation The end of project evaluation will be national, covering all the country. Consideration for rural and urban areas will be made in addition to the agro ecological zones. The evaluation will essentially cover the following thematic areas: i. Project impacts what outputs have been delivered, what changes (outcomes, both intended and unintended) have occurred among the beneficiaries, relevance of project interventions, return on investments and the sustainability of the achievement to-date; 2

3 ii. Project delivery mechanisms the approach adopted by the project to achieve the results highlighted above including models/best practices employed by the project; e.g. linkage of smallholder farmers with structured markets, etc. iii. Project management for results- This would look at the internal program management processes to assess their adequacy, efficiency and effectiveness. These will include the project structure; staffing; processes; level of integration and alignment with other Government programs; and how the project has supported public and private partnerships to facilitate productivity and market access for smallholders. However, the consultant is expected to conduct a comprehensive evaluation. Therefore a detailed list of evaluation questions in relation to appropriateness/relevance, effectiveness, efficiency, outcome/impact as well as sustainability should be included in the inception report. 4.0 Methodology While a detailed survey design and methodology to achieve the objectives of the program evaluation shall be defined by the consultant, and approved by the Client, the following is broadly suggested as a guide. The Evaluation methodology will be a combination of qualitative and quantitative approaches for data collection and analysis. Essentially, this will entail: Desk review of key program documents and program implementation progress reports program officers and other partners. Such documents shall include but not limited to the background project documents; ATAAS Project proposal; progress reports; projects rapid assessment reports, and other key documents related to the project. Besides the review of relevant literature related to the assignment, the consultants shall also undertake field data collection using checklist to undertake key informants, notably; farmer organizations, agro-dealer networks, project administrative and technical personnel, project partners/and donors, ATAAS project staff and management; and targeted Focus Group Discussions (FGDs), as appropriate. The main evaluative evidence will be obtained from detailed original review of the implementation and outputs of the project. This will encompass an analysis of existing data (administrative, operational and financial), interviews and discussions with key officials at MAAIF & NARO and respective implementing institutions, field visits to selected number of randomly selected sites, discussions with key stakeholders at national, district, sub-county and field levels. To enrich the qualitative data obtained through the above qualitative methods, the Consultants shall also administer questionnaires to a representative sample size of farmer households and other relevant stakeholders. To enhance likelihood of achieving the evaluation objectives stated above, the consultants shall work closely with the M&E staff to ensure effective coverage of the project areas, agricultural zones/ ecological zones and cropping systems. It is expected that the consultants will develop and institute an in-field quality control 3

4 mechanism to ensure robust and valid data is collected and analysed for reporting. The sample selection for the household survey will be in cognisance of the target commodities and the agro-ecological zoning. A multistage random sampling design is proposed. A representative sample will be designed to allow for reliable estimation of indicators of interest (as in the Results Framework) at the national, rural-urban, regional levels and for the Agro Ecological zones. A control group is also proposed. The findings so generated will be complemented with qualitative analysis based on focus group discussions with main categories of participants in the ATAAS Project as well as beneficiary and non-beneficiary farmers assessments to document the implementation of the project, client satisfaction with the project, its processes, and intermediate outcomes, outcomes, and impacts. A more detailed and/or revised methodology is expected of the consultant (s), at the inception of the assignment. The detailed methodology to be used for evaluation is expected to be outlined by the Consultants in the Inception Report. In order to objectively achieve the purpose above, the evaluation methodology will be guided by the following ethical rules/considerations: i. Openness of information given, to the highest possible degree to all involved parties ii. Broad participation the key stakeholders should be involved when iii. relevant/possible Reliability and independence the evaluation should be conducted so that findings and conclusions are correct and trustworthy. Therefore, the the consultant should conduct the study in accordance to the standards for effective evaluation (Joint Committee on Standards for Educational Evaluation, 1994) about utility, feasibility, propriety, and accuracy as well as the guiding principles for evaluators (American Evaluation Association, 2004) about systematic inquiry, competence, integrity/honesty, respect for people, and responsibilities for general and public welfare. 5.0 Duration of the assignment The assignment is expected to take 14 from the date of signing the contract. 6.0 Expected Outputs and Timelines The suggested timetable for the deliverables is as follows: Sr. # Item Content Time after contract signing 1 Inception Report The Inception report shall include the detailed proposed methodology, implementation plan, clear timelines, key tasks, and deliverables. The 4 Two (2)

5 Sr. # Item Content Time after contract signing Consultant will be required to present the inception report to MAAIF and NARO for review and approval. 2 Draft ATAAS Project Evaluation Report in hard and soft copies 3 Final ATAAS Project Evaluation Report in hard and soft copies 4. Report presentation On completion of data collection and analysis, the consultant will immediately draft the ATAAS end of Project Evaluation report. The report should consist of Background of the consultancy, Executive summary, including recommendations, Table of contents, Project Goals, Evaluation objectives, Methodology: scope and coverage, evaluation design and sample size, organization, quality control, data processing and management, funding, level of reliability of the estimates of the indicators being evaluated, Main findings, Conclusions and recommendations, Appendices to include TOR, list of references etc. The Consultant will present 4 bound hard copies of the draft report and a CD. The Final Report should incorporate comments and suggestions from MAAIF and NARO, and other stakeholders, as well as any other data not fully captured at the time of writing the draft report. The report should consist of Background of the consultancy, Executive summary, including recommendations, Table of contents, Project Goals, Evaluation objectives, Methodology: scope and coverage, evaluation design and sample size, organization, quality control, data processing and management, funding, level of reliability of the estimates of the indicators being evaluated, Main findings, Conclusions and recommendations, and appendices as appropriate. The Consultant will present 4 bound hard copies of the draft report and a CD The Consultant will be required to make a formal presentation of the final report to the Client. 5. Data sets Raw data sets captured with an agreed computer package on a CD Twelve (12) Fourteen (14) Fourteen (14) Fourteen (14) 5

6 7.0 Qualifications and Competence Requirements The Consultant should have a minimum of a Masters qualification in Agricultural Economics, Statistics, Economics or Social Sciences with at least 10 years of professional experience in agricultural development. Competences i. Experience in designing and carrying out Research, Monitoring and Evaluation ii. Working knowledge of Agricultural Extension and Agribusiness Development Services iii. Working knowledge of Agricultural research, technology development and dissemination iv. Working knowledge of Uganda s Agricultural Policies v. Knowledge of World Bank operational policies and procedures relating to Project Management vi. Strong written and verbal communication skills vii. Possess good coordination skills 8.0 Proposed Payments Payments will be made as per schedule of deliverables below; 40% on presentation of the inception report; 30% on presentation of the Draft evaluation report and; 40% on presentation of final evaluation report. 9.0 Roles of the Recipient The recipient, and Client, of the consultancy service is the Agricultural Technology and Agribusiness Advisory Services (ATAAS) Project of the Ministry of Agriculture, Animal Industry and Fisheries. The Client will play the following roles: i. Provide all relevant documentation to the consultant, such as the Project Appraisal Document, Periodic reports, Implementation Manual, etc.; ii. Facilitate review and dissemination meetings with key stakeholders; iii. Provide the Consultant with list and contacts of key stakeholders; iv. Provide the Consultant with introductory letters to key stakeholders to facilitate communication; v. Both MAAIF and NARO shall provide Project Technical staff who will be the Focal persons to facilitate liaison between the Consultant and the Client; vi. Prepare and make timely payments to the Consultant; and vii. Observe the timelines of activities as will be agreed upon in the work plan viii. Validation and approval of the final report Coordination and reporting 6

7 To ensure consistency, enhance utilization and avoid any surprises, the firm (consultant) shall in the course of the evaluation, be required to regularly provide an update on the evaluation progress. Reporting shall be directly to the Permanent Secretary, MAAIF through the Project Task Manager. The Project Monitoring and Evaluation Specialist, on behalf of the Client, shall manage the day-to-day coordination of the consultancy, whilst the Consultant will also appoint an individual as the counterpart. The two will be the points of contact for coordinating logistics and administrative details for such events as the consultative and dissemination meetings. 7