Monitoring and Evaluation for AKIS Projects

Transcription

1 AGRICULTURAL KNOWLEDGE & INFORMATION SYSTEMS (AKIS) GOOD PRACTICE NOTE Monitoring and Evaluation for AKIS Projects Framework and Options Prepared by Gary Alex and Derek Byerlee with inputs from the AKIS Thematic Team December 2000 Work in progress for public discussion The World Bank Rural Development Family Agricultural Knowledge & Information Systems (AKIS)

2 Agricultural Knowledge & Information Systems is a thematic team focusing on agricultural extension, education, and research within the Rural Development Department of the Environmentally & Socially Sustainable Development Network of the World Bank.

3 Contents Foreword Abbreviations iv v Executive Summary 1 I. Monitoring and Evaluation for AKIS Projects 2 II. Challenges for M&E Systems in AKIS Programs 4 Problems Inherent in Technology Program M&E 4 M&E in AKIS Projects 5 III. The Framework for M&E Systems 6 The Logical Framework (Logframe) 6 Impact Models for AKIS Sub-Systems 7 Impact Model for Agricultural Technology Funds 9 IV. AKIS Performance Indicators 9 V. Data Sources and Collection 15 Project/Program Data Management Information Systems 15 National Level Databases 17 Special Studies 17 VI. M&E in the Bank Project Cycle 18 VII. Recommendations 21 A. Describe Details of M&E System in Project Appraisal Document 21 B. Request M&E Data from the Early Stages of a Project 21 C. Develop Permanent M&E Capacity for Implementing Institutions 22 D. Establish A Comprehensive Set of Key Performance Indicators 22 E. Establish Project Baseline Data 22 F. Avoid Large Surveys 22 G. Use a Combination of Data Collection Methods 22 H. Ensure Utilization of M&E Data 23 VIII. References and Additional Readings 23 Annex A. Illustrative Indicators for AKIS Projects 25 Annex B. Example of Routine Project Report Format: Key Indicators for PRODETAB (Brazil) 34 iii

4 Foreword This AKIS Good Practice Note summarizes current thinking and good practice with monitoring and evaluation of agricultural technology projects and programs. The importance of good monitoring and evaluation (M&E) is widely recognized, but establishing M&E systems for research and extension is difficult, and World Bank projects have lagged in introducing sound M&E arrangements for projects and for national institutions. AKIS Good Practice Notes are designed to disseminate views, experiences, and ideas that may assist World Bank Task Team Leaders, national counterparts from Borrower countries, and other partners to prepare and implement projects to strengthen agricultural research, extension, and education programs. The Good Practice Notes contain valuable information about lessons learned from innovative experiences in World Bank projects and elsewhere, and make this information readily available for comment and use by project teams. The Agricultural Knowledge and Information Systems (AKIS) Thematic Team includes World Bank staff working in or interested in research, extension, and education programs. The team objective is to enhance the effectiveness of Bank support to agricultural knowledge and information system development. This, in turn, contributes to the Bank s objectives of alleviating poverty, ensuring food security, and improving sustainable management of natural resources. The AKIS team emphasizes policy, institutional, and management issues associated with agricultural research, extension, and education. Other thematic teams focus on technical issues. The team mission is to promote the development of sustainable and productive agricultural research, extension, and education systems in Bank client countries. This AKIS Good Practice Note was prepared by Gary Alex and Derek Byerlee with input from Jacob Kampen, Dely Gapasin, Madhur Gautam, Matt McMahon, and other AKIS Thematic Group members. Marie-Hélène Collion Chair, AKIS Thematic Team iv

5 Abbreviations AKIS APL ATF CAS CDB ER ICR INFORM IPR ISNAR KPI LIL M&E MIS OED PAD PDB SAR SS TTL Agricultural Knowledge and Information Systems Adaptable program lending (or adaptable program loan) Agricultural technology fund World Bank Country Assistance Strategy Country database Expert review Implementation Completion Report ISNAR s Information Management System for Agricultural Research Intellectual Property Rights International Service for National Agricultural Research Key performance indicator Learning and Innovation Loan Monitoring and evaluation Management information system World Bank Operations Evaluation Department World Bank Project Appraisal Document Project database World Bank Staff Appraisal Report Special study Task team leader (for World Bank project) v

6

7 Executive Summary Monitoring and evaluation (M&E) systems have been weak in World Bank Agricultural Knowledge and Information System (AKIS) projects, both for the projects themselves and for the AKIS programs that they support. Increasingly scarce resources for research and extension dictate a need to maximize efficiency in use of public investment in these programs. Improving monitoring and evaluation systems for agricultural research, extension, and education programs is a long-term process critical to improving program management and sustainability, and to building effective institutions to promote economic growth, reduce poverty, and conserve environmental resources. Principles for effective M&E are the same for Bank-financed projects and for ongoing research, extension, and education programs within AKIS institutions. M&E should be based on an impact hypothesis linking activities to desired outcomes and impacts. This hypothesis is reflected in the logical framework (logframe), as used in design of Bank projects. To establish a sound basis for project M&E, Project Appraisal Documents (PADs) must describe M&E systems in adequate detail, addressing questions of what information is to be collected, how (using what procedures), by whom, when, where, and why (how it will be used). The M&E plan should describe arrangements for obtaining baseline or control data; assess capacity for carrying out M&E; define indicators and targets; identify investments needed to strengthen M&E capabilities of implementing agencies; and identify key assumptions or issues to be addressed in project evaluations (the Mid-Term Review). Other good practices for M&E of AKIS projects and programs include: Requiring M&E data from the early stages of a project: The M&E framework should be established and data collection begun at the outset of a project or program. It is difficult to retrofit an M&E system without compromising quality, objectivity, and usefulness later, when data are needed for management or evaluation. Institutionalizing M&E capacity within implementing institutions: Sustainable AKIS programs and institutions must have M&E systems to improve program management and to demonstrate impacts that justify program funding. Establishing M&E capacity within the institution provides for most M&E needs for Bank projects and for future needs of the institution. Establishing a comprehensive set of key performance indicators: Project M&E should be based on a set of hierarchically linked key performance indicators for inputs-outputsoutcomes, as reflected in the logframe. Targets for key indicators should be defined by quantity-quality-time (TQQ). Generally a good M&E system requires a more extensive set of indicators than are included in the logframe. Establishing project baseline or control data: Baseline data are important but frequently not available at project start-up. Some pro- 1

8 2 Monitoring and Evaluation for AKIS Projects: Framework and Options vision is needed early in the project to ensure the availability of baseline or control data for comparison of with-project and without-project scenarios. Avoiding large surveys: Large-scale surveys are time-consuming and costly and methodology variations between surveys often limit comparability. Large sample surveys are often of limited value for Bank projects, but may be more useful for national AKIS programs that have a longer time perspective. Using a combination of data collection methods: M&E for AKIS projects generally must integrate data from various sources, including the project s internal management information system, special studies (often carried out by independent institutions), and national data sets. Disaggregating data whenever possible by gender, income level, ethnicity, and other key characteristics of client groups: AKIS M&E systems must be able to tell who is benefiting from a program as well as what the benefits are, and projects must be able to document impacts on women, the poor, and minority groups. Ensuring utilization of M&E data by decisionmakers: M&E systems must include strategies for use of data, including reporting formats and schedules designed to be useful to program managers and financiers. Keep it simple! This is the cardinal rule for M&E systems. M&E arrangements must be implementable and feasible within the financial and human resources available. Often the more elaborate the system and data requirements, the less likely it is that the system will be able to deliver on its promises. I. Monitoring and Evaluation for AKIS Projects This AKIS Note summarizes experience and current thinking on monitoring and evaluation systems for AKIS projects. The objective of the Note is to assist World Bank Task Team Leaders (TTLs) and Borrowers to develop practical and effective systems to monitor and evaluate project performance and impacts. Monitoring and evaluation (M&E) systems are required for all Bank projects (Casley and Kumar 1987; World Bank 1989), and are receiving increased attention now that all projects are being asked to demonstrate results. Agricultural sector investments, and particularly research and extension programs, are being questioned as to impact on social indicators. Lack of consistency in definitions complicates understanding of monitoring and evaluation. 1 Monitoring may refer to: routine collection of information, tracking implementation progress, measuring efficiency, and questioning whether the project is doing things right. Evaluation is generally defined in terms of: analyzing information, ex-post assessment of effectiveness and impact, confirming project expectations, mea- 1 The World Bank has defined monitoring as the continuous assessment of project implementation in relation to agreed schedules and of use of inputs, infrastructure, and services by project beneficiaries, and evaluation as periodic assessment of the relevance, performance, efficiency, and impact (both expected and unexpected) of the project in relation to stated objectives (World Bank 1989).

9 Monitoring and Evaluation for AKIS Projects 3 suring impacts, and questioning whether the project is doing the right things. Monitoring (in the sense of continuous collection of data on a project) is required both for implementation and ex-post assessment, whereas evaluation (in the sense of making judgments and decisions about a project) is also a continuous process during project implementation and after completion. In practice monitoring and evaluation (M&E)are difficult to separate and M&E is best defined as a continuous process of collection, analysis, and use of data. Effective M&E systems are important to enable project management to know whether implementation is going as planned, and to provide information needed to adjust implementation plans to achieve desired results. M&E systems are also of critical importance to provide evidence of project accomplishments and to defend project funding requests. By supporting implementation and budget requests, sound M&E systems contribute substantially to enhancing program sustainability, and are neces- sary elements of research, extension, and agricultural education programs. M&E systems should aid in assessing two aspects of a project performance and results. M&E of project performance essentially assesses project implementation efficiency, whereas M&E of project impact assesses project effectiveness. Project performance (process) leads to results (impacts). This distinction between performance and impact is not entirely clear as there are often intermediate impacts (or results) between project performance and impacts. Despite its importance, few World Bank-financed AKIS projects have adequate M&E systems (see Box 1). M&E plans are sometimes prepared as last-minute additions to satisfy project preparation requirements and many current M&E systems contribute little to project management and performance assessment. Project M&E systems must be comprehensive, well planned, and adequately funded. This can require substantial funding, which reduces funds available for program implementation. Box 1. Weaknesses in AKIS Project M&E Plans A desk review of 45 appraisal documents (SARs/ PADs) for AKIS projects financed from 1991 to 1999 revealed that only 24 percent had adequate M&E plans. Most appraisal documents included a separate M&E section (78 percent), specified information to be collected (78 percent), and provided financing to strengthen national program M&E systems (62 percent). The number of M&E plans judged adequate increased from 14 percent of 21 projects financed during to 33 percent of 24 projects during The quality of key performance indicators showed little change between these periods. Even for projects financed from 1994 to 1999, only 42 percent had well-defined and quantified indicators for outputs, and 17 percent for outcomes. A key informant survey of TTLs for 18 AKIS projects confirmed the weaknesses in M&E systems for research and extension projects. Most projects have input monitoring systems to track financial flows and physical progress; some track outputs, but few have well-defined outcome or impact indicators. Typically, indicators and monitoring systems are developed on an ad hoc basis late in the project (for example, at the Mid-Term Review), when decisionmakers expect evidence of progress and program results. In some cases, projects collect large amounts of data, but little of this is analyzed or used. The basic problem is lack of demand for M&E data from policymakers and managers (including World Bank management). TTLs are almost unanimously concerned with M&E, and most projects are actively working to establish M&E systems and local capacities. TTLs recognize that demands for evidence of program impacts will not go away, and that research and extension programs must demonstrate a plausible link to impacts on poverty and other social objectives. A clear lesson from the survey is that projects must establish their M&E systems early, and develop strategies to increase use of M&E data for program management and constituency building. Source: Based on Report for AKIS Retreat 2000 by Madhur Gautam and Gary Alex.

10 4 Monitoring and Evaluation for AKIS Projects: Framework and Options One rule of thumb used by some agencies is that M&E costs should generally not exceed five percent of program costs. There is always a tension between M&E data needs and acceptable levels of funding. Compromise is necessary and this AKIS Note aims to provide practical guidelines to improve project M&E. M&E systems are essential to Bank projects, because of the Bank s fiduciary responsibilities. They are also needed as permanent elements within country technology programs, and AKIS projects with institution-building objectives must address the issue of establishing permanent M&E systems. Although the M&E needs of a Bank-financed project and a national technology system are similar, they are not necessarily identical, and a distinction often must be recognized in M&E system design. In practice, principles of M&E are the same whether for a Bank-financed project, a national institution, or a program financed within a national technology system (for example, an extension program or a livestock research program). In discussing M&E systems, this Note will use the terms project and program interchangeably, and, when referring to a specific situation, will specify applicability to Bank project M&E or national program M&E. II. Challenges for M&E Systems in AKIS Programs Monitoring and evaluating project performance and impact are difficult exercises for most development programs, but pose special problems in World Bank-financed AKIS projects. Problems Inherent in Technology Program M&E Assessing technology program performance and impact is inherently difficult. Research outcomes and technology adoption are by nature uncertain, and establishing a timetable for results is difficult. Technology development and dissemination do not follow a linear path the environment for innovation changes continuously with new research results or changes in institutional, market, or social conditions. Even with success, lag times for impact of agricultural research are long, with applied research often requiring years to produce full impacts on productivity. Technology programs increasingly target the diverse objectives of reducing poverty, conserving environmental resources, promoting food security, increasing exports, increasing economic growth, and reducing social conflict. It is difficult to measure some of these, and trade-offs between objectives are often necessary. Cause-and-effect attribution of impact is problematic because poverty, economic growth, and environment are affected by weather, politics, policies, institutional changes, and many other factors. As well, different agencies are involved in basic, strategic, applied, and adaptive research, extension, and technology adoption. Program performance and impact depend on coordination between technology institutions and input supply systems, rural credit, product marketing systems, infrastructure development, trade and macroeconomic policies, and other factors. Technology program funding is in decline, and institutions must limit investment in M&E even though collection of reliable data can be costly. Furthermore, lack of reliable national

11 Challenges for M&E Systems in AKIS Programs 5 data is a serious obstacle in many countries. In addition, lack of awareness and managerial demand for M&E data is an overriding problem that inhibits development of M&E systems. As most countries lack capacity for M&E, decisionmakers have low expectations for M&E information. This leads to neglect of systems for generating information, and a tolerance for poor performance of the systems. Donors have often been the only supporters of M&E, but as technology systems mature, funding groups (ministries, donors, or the private sector) will demand more effective program M&E. M&E in AKIS Projects AKIS projects face two apparent dilemmas in planning M&E systems. The first is whether to develop an M&E system specific to the needs of the Bank project, or to the broader needs of the national research, extension, or agricultural education program. Both are needed and, though similar, each has somewhat different requirements. Bank project M&E systems focus specifically on project implementation and outcomes, even though this is usually only one part of the national technology system. This is, however, must-have information for the TTL and as a minimum Bank projects must establish appropriate M&E systems to cover the activities under the project. National technology programs need M&E systems that are comprehensive and not focused solely on Bank-funded activities. These systems enable an institution to focus on results, learn from experience, and enhance institutional sustainability. 2 National program M&E systems provide information needed to set priorities, plan program implementation, and defend budgets, work plans, and strategies. The systems should therefore be flexible and able to assess performance and impact of the institution as a whole, of individual programs within the institution, or individual projects under a program. In practice, Bank projects are usually able to address most project M&E needs through an M&E system established by the implementing institution with overall responsibility for the Bank loan. Such a system is often an appropriate or necessary component of a Bank project, as the implementing institution is responsible for public funding and must have an M&E system adequate to manage these funds. This may require a centralized M&E unit within the main implementing institution (the Ministry of Agriculture) collaborating with M&E units in other co-implementing institutions (extension agencies), and in decentralized regions where project activities take place or have influence. Institutionalizing an M&E system within implementing institutions is key, as this will then meet Bank needs, as well as provide a permanent M&E system beyond the life of the Bank project. Effective M&E systems should encourage an M&E mentality throughout the technology system. The second apparent dilemma relates to project objectives. AKIS projects typically seek productivity impacts 3 (often occurring over a long period of time), but these usually require institutional development to bring about the productivity impacts, and to provide a basis for continued innovation and sustainable impact on productivity. Should project M&E focus on the institutional development or on the productivity impacts? Institutional development without productivity impact does not justify investment, but neither is an investment sound if it generates a quick productivity impact without the institutional development necessary to sustain it over time. 2 Current strategies emphasizing institutional pluralism in technology systems result in additional complications, because all activities are not under one institution and each institution may require its own M&E system. 3 Productivity impact is defined broadly to include any impact on the agricultural production system and rural development, including environmental, economic, and social welfare.

12 6 Monitoring and Evaluation for AKIS Projects: Framework and Options Most Bank-financed projects combine both institutional and productivity impacts, though with the balance varying from project to project. Bank-financed projects must recognize these two types of impact, define Project Development Objectives in light of individual country and program circumstances, and design M&E systems that capture data relevant to the principal project objectives. 4 Adaptable Program Lending (APL) offers an option for handling this problem in some cases, by emphasizing institutional development objectives (and M&E) in earlier phased projects, and productivity impacts (and M&E) in later projects. III. The Framework for M&E Systems All projects are essentially an investment plan based on an hypothesis of how project-financed activities will lead to desired outcomes and changes in social and economic conditions. This hypothesis provides the basis for project monitoring and evaluation. The Logical Framework (Logframe) The sequential steps of expected cause-andeffect due to project activities constitute the project logic (or the impact model ). For AKIS projects, this can be represented in a multi-step causal hypothesis as illustrated in Figure 1. For bureaucratic consistency in World Bank and most other project systems, this multi-step process by which investments lead to impacts is compressed into a four-step format in the logical framework (logframe) (see Table 1). The logframe represents the project hypothesis of activity-result relationships, and is the basis for project design and evaluation (World Bank 1996). However, it may be useful when devel- Figure 1 Hypothesis of AKIS Project Impact Process Agricultural Education and Institutional Development Knowledge Generation Knowledge Dissemination Opinion Change Knowledge Change Practice Change Productivity Change Change in Social Goals Note: This simplified diagram omits important feedback loops that regulate and strengthen the technology generation and dissemination process. 4 It is in Bank-financed institutional development activities that M&E for Bank-financed projects may differ most markedly from M&E for the national technology program. Some indicators (for example, level of stakeholder participation in priority setting, share of private sector financing for services, and number of collaborative research programs) may be a high priority for understanding impacts on institutional development, but less important for the national system s routine M&E programs.

13 The Framework for M&E Systems 7 Table 1 The World Bank s Logical Framework Narrative Summary Sector-related goal Project development objective Outputs Project components/ sub-components Key Performance Indicators Inputs: (budget for each component) Monitoring & Evaluation Plan Critical Assumptions (Goal to Bank Mission) (Objective to Goal) oping a project M&E system to consider the full chain of causality, and not be constrained by the four-step format of the logframe. Logframe outputs and objectives should appropriately address equity issues relating to gender, ethnicity, and poverty group. Key performance indicators should disaggregate data on the same basis to encourage equity-sensitive project implementation, monitoring, and evaluation. Guidelines for such indicators can be found in Gender-sensitive Monitoring and Evaluation for Rural Development Projects (World Bank 2000). Impact Models for AKIS Sub-Systems Extension programs are based on a project impact hypothesis of: extension activities, increasing extension service contact with farmers, changing farmers opinions, changing farmers knowledge, increasing adoption of new technologies, changing productivity, and changing social indicators. Extension activities that target particular crops or production systems may have close, well-defined links to impacts on productivity and social goals, but even in these programs M&E is complicated by the substantial lag between extension activity and productivity impacts and by influence of external factors. In extension programs that serve as general rural information systems, or respond to diverse production and social problems with no defined commodity, program M&E is further complicated because impacts may be diverse and difficult to measure. Research program performance and impact assessment is, perhaps, even more challenging, due to the nature of research, the longer lag times involved, and the more complex chains of cause and effect. Technology development draws on input from various stages of research (basic, strategic, applied, adaptive), and from producers own inputs from traditional knowledge and technology adaptation. Impacts from research accrue and technologies are proven useful only later, following technology dissemination and adoption by farmers. However, methodologies for estimating productivity impacts are well developed. ISNAR has done extensive work on methods for M&E in research organizations (Ballentyne and others 1993; ISNAR 2000). Agricultural education is even further removed from direct impacts on productivity. Agricultural education investments lead to more (or better) trained individuals, who are employed in positions in which their skills enable them to perform better. Their improved individual performance leads to better performance of research, extension, and other programs, with a consequent improvement in agricultural productivity. For agricultural education, project performance M&E is fairly straightforward for training and institutional development, but education impacts on social goals are loosely linked, diffuse, and difficult to estimate.

14 8 Monitoring and Evaluation for AKIS Projects: Framework and Options For any AKIS program, an M&E system to assess project progress at different levels of the logframe (project cause-and-effect hypothesis) requires a range of monitoring tools, many already in use and available to be applied on a systematic basis (Table 2). A comprehensive approach to AKIS program M&E should: be based on a logical framework of cause-andeffect relations that conceptually relate programs to impacts on targeted social goals; address both project performance and impact; utilize different tools for monitoring different stages of the technology generation, adoption, and impact process; Table 2 Methods for M&E for Assessment of AKIS Projects Program Level Stage of Activity Monitoring Indicators Method of Assessment Budget Funding Funding Budget and ex ante impact analysis for initial budget Inputs Procurement Input quantities Program accounting and financial audit Outputs Extension execution Outcomes (Project Development Objective) Impacts (Goal) Research execution Agricultural education execution Technology adoption Institutional development Contacts with farmers; opinion change; and knowledge change Knowledge generation; technologies developed; recommendations released; research milestones Graduates and competence of graduates Technology adoption and initial impacts Institutional productivity (i.e., numbers of extension contacts or technologies developed) MIS data; case studies, surveys, or beneficiary assessments MIS data; external reviews complemented by ex ante impact analysis and output metrics MIS data; institutional reports and case studies or surveys of graduates and employers Special studies; MIS data Special studies; MIS data Impact Social goals National and global data; modeling Probable Frequency of Assessment At initial project approval and then annually Annually Routinely, at least annually Variable: perhaps every 3-5 years Variable: annual to every 3-5 years Variable: perhaps through infrequent studies, often requiring baseline data Variable: perhaps through infrequent studies, often requiring baseline data Regular collection of data with impact expected only over a number of years. Source: Adapted from Alex (1998).

15 The Framework for M&E Systems 9 relate project progress and results to planned targets based on ex-ante projections of impact; establish a management information system for regular reporting on inputs and outputs and, when possible, on outcomes; utilize special studies for monitoring and evaluation of output quality and outcomes; utilize currently available information and review processes to keep costs down and remain sustainable; and include mechanisms to feed information back to decisionmakers. Impact Model for Agricultural Technology Funds Agricultural Technology Funds (ATFs) and other projects that have multiple subprojects illustrate another level of complexity in assessing performance and impact of technology system programs. These projects have impacts both on direct program participants (beneficiaries) and spillover effects on indirect beneficiaries. ( Beneficiaries presuming that the indirect effects are positive.) Spillovers on indirect beneficiaries are difficult to measure. In addition, ATFs generally also promote institutional change within the technology system, change which itself leads to indirect impacts of the pro- gram. A model of ATF impacts is illustrated in Figure 2. Comprehensive assessment of performance and impacts requires measurement of impacts A, B, C, and D in Figure 2, as well as measurement of the performance efficiency (in terms of inputs and outputs) of the ATF program and its individual subprojects. In practice, measurement of impacts at A is relatively straightforward; B and C are much more difficult; and D is nearly impossible to measure. Figure 2 Impact Model of an Agricultural Technology Fund A Direct Sub- Project Impacts Competitive Grants Program C B D Indirect Impacts Institutional Change IV. AKIS Performance Indicators Performance indicators (or KPI key performance indicators) are the basis for project or program M&E. Performance indicators derive from logframes and provide for performance measurement at two levels: process indicators that measure program inputs and outputs, and impact indicators that measure outcomes (changes related to the Project Development Objective ) and impacts on social goals. Tracking input use and direct project accomplishments often requires multiple indicators, but these are relatively easily measured. Outcome

16 10 Monitoring and Evaluation for AKIS Projects: Framework and Options Box 2 Sample Key Performance Indicators Research institution development project: Examples of possible indicators for an extension project and an institutional development research project are listed below. Production-oriented extension project: 35 farmer training programs are held each year for five years with each course including practical demonstrations and having at least 50 participants (Input level). Number of farmers trained and know how to use organic fertilizers in conjunction with chemical fertilizer: 10,000 by end of year three and 20,000 by the end of year five (Output level). Number of farmers using organic fertilizers in conjunction with chemical fertilizer: 20,000 by beginning of year three and 40,000 by the beginning of year five (Outcome level). Within 10 years, net farm income in the region has increased by an average of 20 percent (Goal level). 20 scientists trained to the M.Sc. level in year 3, 40 in year 4, and 15 in year 5. (Input level). By the end of year 2, six provincial research centers have completed strategic plans based on sound social and economic analysis; by end of year 4, at least 12 collaborative research programs are being executed with collaborating institutions and in line with strategic plans; and by year 5, 80 percent of research programs have at least 50 percent of trials carried out in collaboration with small farmers (Output level). By the end of year five, at least 12 research programs are receiving at least 25 percent of their funding from the private sector, are rated satisfactory or better by external peer reviews, and are producing technical recommendations with demonstrated potential to increase small-farm profitability (Outcome level). Within 10 years, net farm incomes increase by an average of 20 percent due to adoption of new technologies (Goal level). and impact indicators are more difficult, as they are a result of project activities, but are influenced by many other factors. Project indicators will vary depending on specific project strategies and objectives. Sound indicators must be characterized by time, quantity, and quality (TQQ). They should specify when change in the indicators is expected to be achieved, the extent of change expected, and the quality standard for the expected change. Qualitative indicators are acceptable, but must be specific enough to be unambiguous as to whether or not they are achieved. General principles for selecting key performance indicators require that they be: Relevant to the development objective of the project; Unambiguous as to definition, valid and specific in reflecting changing conditions, and consistent and verifiable in measurement; Meaningful and of interest to project management and stakeholders; Significant and selective in that the monitoring system should limit the number of indicators to a manageable number; Practical in terms of the ability to collect data in a timely and cost-effective basis and with a reasonable degree of reliability; Provide information on impact disaggregated by gender, poverty status, ethnicity, or other characteristic of client group; Acceptable to the Borrower, the Bank, and other stakeholders; Quantifiable to the extent possible, or, if qualitative, based on a common understanding of what constitutes success; and Responsive in reflecting changes due to project activities and sensitive in demonstrating change in as short a time frame as possible. Budget and Input Level. Technology institutions need well established systems for budgeting, procurement, and audits. These systems track funding allocation, input (works, goods,

17 AKIS Performance Indicators 11 services) use, and expenditures by program and project. Reporting systems should provide regular information on cost and input allocation to specific programs, projects, and cost centers. For research programs, management software packages, such as ISNAR s INFORM program (ISNAR 1991), provide research cost data in detail by project and program. These can be useful, but may be overly complex for many research institutions, so less formal methods of estimating expenditures by project may be fully acceptable. Simple, bare bones systems are needed. The planning process establishes budgets and input requirements as part of priority setting. These cost projections (budgets) must be linked to expected outcomes and are an essential element for evaluation of any program, as benefits are assessed in relation to the program cost. Accurate and timely budget and expenditure data are important to any evaluation of program efficiency and effectiveness. Expenditure data are commonly used and helpful in tracking implementation. If used alone, however, it is an inappropriate and insufficient indicator of project implementation progress and performance. Ex ante economic analysis relating project costs and benefits is a standard means of evaluating proposed investments, and is therefore relevant to evaluating investment budgets; ex post economic analysis uses similar methodology to evaluate investment impacts (see Box 3). Output Level. AKIS projects typically focus on two objectives: institutional development to strengthen the institutions and technology system necessary to develop and disseminate improved technology and management in the agricultural sector; and productivity change due to technological innovation introduced through the technology system. The two objectives are related and mutually supportive, but require different indicators for outputs and outcomes. An illustrative list of key performance indicators is included in Annex A for the two broad types of AKIS project objectives for research, extension, and education. Project outputs are direct products of project activity and are within the control of the project. Project output level M&E combines relatively simple quantitative measures of productivity with more complex assessments of the quality of the outputs produced. Data on outputs should come from management information systems (MISs), which provide routine reporting against expected output targets established during project planning. Reporting systems should be structured to provide this information on a regular basis to guide program man- Box 3 Economic Analysis of AKIS Projects Comprehensive economic analysis of AKIS projects that includes major institutional development components is generally not very meaningful or helpful in evaluating investments (Horstkotte-Wesseler and others 2000). In place of a comprehensive economic analysis, good practice in AKIS project preparation should include: Evaluating research investments at the level of the research program, without limitation as to whether the source of financing is from Bank loans; Using cost effectiveness analyses for evaluating some extension investments and most agricultural education investments; Using break-even analysis to evaluate extension programs or projects proposed for Bank financing; Using economic surplus, or modified economic surplus, analyses for economic analysis of technology programs; Using ex ante economic analyses to aid in priority setting and planning within the national technology system; and Building economic analysis capability within national programs to provide a basis for ongoing economic analysis of technology programs. Maredia, Byerlee, and Anderson (2000) review procedures for ex post economic analysis of research programs. Their paper provides an overview of current practice and experience, summarizes and clarifies methodological issues, and identifies guidelines for good practice in impact evaluation.

18 12 Monitoring and Evaluation for AKIS Projects: Framework and Options agement. Qualitative assessment of outputs requires special studies and independent data collection, which should be built in as a regular part of program M&E. For extension projects with institutional development objectives, output M&E is based on physical completion (for example, staff trained, extension agents posted) and qualitative assessment of established capacities. For productivity-oriented projects, output M&E relies on data from program reports, especially in terms of deliverables in extension system contacts with farmers (for example, numbers of farmers trained, farmer questions answered, field day attendance, newspaper or magazine readership, radio campaign listeners). Misra (1997) describes approaches and indicators for M&E of extension programs. Quantitative measures of interaction with farmers then need to be supplemented by quality measures, such as subjective program assessments or measures of the program s ability to promote adoption of new technology (behavior change). Indicators of potential to influence behavior (for example, attitudes towards extension services, change in knowledge about technologies) are usually assessed through field surveys, case studies, or beneficiary assessments. Extension program M&E systems must assess the quality of both the message (the relevance of the technology or advisory services) and the medium (the effectiveness of the communications or delivery system). Questions concerning the message are in a sense an assessment of research or technology generation, as well as of the extension service s ability to access relevant technologies. For research programs, output indicators for institutional development projects are also based on physical completion (for example, laboratories constructed, staff trained) and quality of established capacity. Productivity-oriented projects present serious difficulties in developing meaningful indicators, as research activities may be completed as planned, but completion says little about the quality or significance of the work. 1 Resulting increases in knowledge may have no immediate effect outside of the laboratory, but be of great value at some future time. As with extension program M&E, monitoring research outputs requires a combination of output metrics (for example, experiments completed, papers published) and special studies (external peer reviews) to assess research program quality. External peer reviews are useful to obtain technical experts assessment of research activities from three perspectives: a prospective review (to assess implications and relevance of research); an in-process review (to assess efficiency of research and performance against targets for outputs), and a retrospective review (to assess scientific or technical quality of work completed). Agricultural education projects with institutional development objectives rely on physical measurements of outputs (faculty trained, curricula developed) and related special studies to assess quality of the outputs. Developing capacity for assessing increase in numbers of individuals trained is much easier than for assessing capacity for improved quality of training. Agricultural education programs with direct training objectives have outputs that are easily quantified in terms of numbers of graduates or person-months of training. Quality assessment is more difficult, and involves special studies in the form of surveys of change in competencies or knowledge of graduates. Data and special studies evaluating market demand for training (rates of employment of graduates, salary premiums commanded by graduates, or willingness to pay for training) are possible tools for assessing the quality of agricultural education programs. 1 Research impact depends mainly on the results of the best research and relatively little on the total quantity of research.

19 AKIS Performance Indicators 13 Project Development Objective (Outcome) Indicators. Outcome indicators reflect the initial results or intermediate impacts of a project. Outcomes are closely related to project activities, but are not under the full control of the project. The outcome indicators must measure achievement of the project objectives, whether relating to institutional development or to productivity change. Institutional development outcomes vary considerably depending on technology system needs. These may reflect changes in the institutions themselves, institutional productivity, or institutional relations with clients. Changes in institutions may be measured by numbers of functional facilities (for example, laboratories, rural communications centers), number of qualified staff, ratio of scientists to support staff, numbers of research or extension service providers, new institutional relationships, ability to address new problems, peer reviews of technical programs, stakeholder attitudes, shift of funding (for example, to environmental programs or small farmer crops, to competitive funding, to on-farm research), and numbers of collaborative programs. Institutional development indicators reflecting extent and quality of service provision might include publications per scientist, client contacts per extension agent, technology releases per research program, quality rating of competitive grant research programs, and client satisfaction ratings for extension services. Financial performance and sustainability indicators might include percentage of funding from private sector or farmers, cost per farmer extension contact, cost per student graduated, and ratio of salary to operating cost budgets. Productivity outcomes would require similar indicators for both extension and research. Metrics may be fairly straightforward, as initial impacts can be measured in the expected uptake pathways in which effects of technical changes are relatively closely linked to and attributable to research and extension outputs. A wide range of possible indicators can reflect two results of program activities change in behavior (adoption) and initial effects of adoption. Examples of the former include rates of adoption of new technology, changes in government policy based on research findings, or new investment made possible because of new technology or management options. Initial effects of behavior change may include yield increases, decreases in deforestation or erosion, production cost decreases, or (for effects of policy reforms) further change in technology adoption rates or investment. Project outcomes cannot always be reported on a routine basis by a project management information system. For this reason, special studies are usually necessary to measure outcomes. Baseline studies followed by regular longitudinal studies over the course of a project are useful to measure changes in a target region. National surveys may be an ideal, but are generally too costly, require too much time to produce useful data, and are not often recommended. AKIS projects should frequently build routine field case studies into M&E systems for extension and research. In fact, it may be difficult to envision an efficient technology system without a functional system for such studies to assess how well a program is doing and how well technologies are being accepted. A key challenge is that of relating with-project changes to the counterfactual without-project scenario. In addition to surveys or case studies to assess project outcomes, other special studies might be needed to track specific planned and unplanned impacts or program operational issues (for example, technology access by women or minority groups, and changes in land tenure and other social conditions in the area). Beneficiary and stakeholder participation in M&E is critical to ensure that AKIS programs are responding to genuine needs of intended clients. This can be accomplished by using survey techniques or participatory evaluation. One participatory M&E approach is beneficiary assessment, which assesses change in farmer opinions and knowledge, changes in behavior and technology use, and changes in production, incomes, and environmental conditions

20 14 Monitoring and Evaluation for AKIS Projects: Framework and Options (see Box 4). Beneficiary assessment uses informal surveys, structured direct observation, and focus group interviews to elicit information. Informant selection is a critical issue as selection is rarely random and poor sampling can introduce significant bias. Beneficiary assessment can be useful, but it can be difficult so consultation with experienced practitioners is recommended. Sector Goal Indicators. Sector goal indicators measure change in the broad development goal to which the project contributes. Changes in these indicators are indirectly attributable to the project, and usually cannot be measured during the life of a Bank-financed project. Although demonstrating direct project impact on a goal level indicator may not be within the manageable interests of a project, it is important that the project provide a plausible hypothesis of how project activities will lead to change at the level of the sector goal. This often requires other sources of research and analysis to build a plausible link between project activities and the social goal. Goal level indicators that are associated with agricultural technology and appropriate to AKIS projects might include: increased factor productivity; social rate of return to investment; positive trends in environmental indicators; and reduction in poverty rates and improved nutrition. Sector goal-level impact indicators are similar for research, extension, and agricultural education. These are affected by many factors other than AKIS programs, and indicators usually show response only with a considerable lag time. Ex post impact analyses are important to demonstrate past success and guide future investments (see Box 4). The trick with impact assessment is that of comparing the with project situation with the counterfactual without project scenario. 1 Despite the difficulties with attribution and detecting change, AKIS projects should link activities to impacts on indicators of social goals as a basis for justifying investments. Box 4 Beneficiary Assessment Getting the Client s Views Beneficiary assessment involves a process of information gathering to assess the value of an activity as perceived by its intended beneficiaries. For AKIS programs, it involves structured conversational interviews with farmers, extension agents, extension managers, agribusiness personnel, and researchers. Extension program beneficiary assessment is based on interviews with a fairly large sample of farmers, stratified as needed by region, gender, ethnicity, exposure to extension services, area coverage by extension programs, or other factors. The interview stresses listening. The purpose of the beneficiary assessment is to influence program policy and program management. A review of experience in ten African extension projects found that the structure for beneficiary assessments varied widely, but that the approach was considered quite effective as a tool for M&E. Farmers in direct contact with extension agents benefited substantially, with resulting impacts on productivity. Farmer-to-farmer technology transfer and service provision to women farmers was generally ineffective. Farmers major criticism of extension was the narrow focus of services. Extension managers found beneficiary assessments to be a positive exercise, effective in introducing policy changes, and resulting in more participatory approaches, including better links to existing and local organizations and greater responsiveness to farmer needs. Source: Salmen (2000). 1 Various methods may be used to establish controls for impact assessment: randomized controls with beneficiaries randomly placed into two groups; constructed controls with participants paired with nonparticipants; statistical controls comparing groups after correcting for other characteristics; reflexive controls comparing before and after; generic controls comparing against norms for change; and shadow controls comparing against estimates of normally expected change.