Global FFS Meeting on Impact Assessment and M&E

Transcription

1 Global FFS Meeting on Impact Assessment and M&E Bangkok, Thailand September 2018

2 Contents Acknowledgments... List of abbreviations and acronyms... Page ii iv 1. Background Workshop... 3 Day 1: Monday, 18 September 2018: Field Visit... 3 Day 2: Tuesday, 19 September Day 3: Wednesday, 20 September Day 4: Thursday, 21 September Conclusion. 32 List of Annexes:... Annex 1: List of Participants... Annex 2: Workshop Programme... Annex 3: Field Visit... Annex 4: Posters... Annex 5: Presentations... Annex 6: Group Sessions... Annex 7: Expectations, Feedback and Final Evaluation i

3 Acknowledgments This report presents the main results of the four-day Global FFS Meeting on Impact Assessment and M&E, held in Bangkok, Thailand, from 17 to 20 September The event was organized by the Food and Agricultural Organization (FAO) of the United Nations and the Thai Education Foundation (TEF). The workshop was planned and prepared by Marjon Fredrix, AlmaLinda Abubakar, Deborah Duveskog, Anne-Sophie Poisot, Suzanne Phillips, Jan Willem Ketelaar and Marut Jatiket. AlmaLinda Abubakar facilitated the workshop sessions. This report was prepared by Gerd Walter-Echols with the help of audio recordings made in the meeting room. The organizers of the workshop would like to express their gratitude to all participants for their poster contributions and active participation in the sessions. Without the inspiring debates, the passionate discussions and innovative visions, this workshop would not have been a success. Thanks also go to the resource persons Henk van den Berg, Gerd Walter-Echols and Kevin D. Gallagher for sharing their experiences and insights. Finally, special thanks go to the local organizing teams from FAO and TEF for making the travel arrangements, organizing the logistics and ensuring the smooth running of the workshop. ii

4 In Memory Nugroho (Nugi) Winarto passed away shortly before the workshop. He was one of the original innovators of Farmer Field Schools in Indonesia. iii

5 List of Abbreviations and Acronyms AESA CIP DOA DOAE EGAD EIQ FAO FAORAP FBS FFS / FS ICT IFAD IIRR IPM IT M&E MEL MOA MFVP NFE NGO ODK PRA PRR RAB SSI SOFT TEF TOF TOT Agro-ecosystem analysis International Potato Center Department of Agriculture Department of Agricultural Extension Intergovernmental Authority on Development Environmental Impact Quotient Food and Agricultural Organization FAO Regional Office for Asia and the Pacific Farmer business school Farmer field school Information and communication technologies International Fund for Agricultural Development International Institute of Rural Reconstruction Integrated Pest Management Information Technology Monitoring and evaluation Monitoring, evaluation and learning Ministry of Agriculture Myanmar Fruit, Flower and Vegetable Producer and Exporter Association Non-formal education Non-government organization Open Data Kit Participatory rural appraisal Pesticide risk reduction Rwanda Agriculture Board Semi-structured interview Society of Facilitators and Trainers Thai Education Foundation Training of facilitators Trainer of trainers iv

6 1. Background The FFS approach is being applied in over 90 countries, in different ecological zones and on a broad range of crops and topics. FAO continues to support FFS programmes, provides expertise, and links communities of practitioners in different regions. However, many practitioners have expressed a need for improved mechanisms to maintain the quality of FFS programmes, which can be eroded because of poor programme design, rapid scaling up, insufficient investment in developing capacity, lack of continued support to FFS facilitators or lack of FFS expertise at critical moments of programme development. With the large number of FFS programmes in different countries, it is a growing challenge to document their lessons-learned and impacts. Information is often difficult to access when it is only available as project documents and reports rather than official publications. In some cases limited efforts are made to maintain databases on FFS programmes and/or impacts of FFS interventions. The methods for documenting changes vary from programme to programme, and impacts have been determined in different areas such as technical, socioeconomic or policy. The Global FFS Platform has made its goal to promote FFS towards a more sustainable and resilient agriculture. It facilitates exchanges, provides tools for improving FFS quality, and - where appropriate - supports further institutionalization of the FFS approach in the strategies and practices of farmer organizations, civil society and governments. Today, the Global FFS Platform is the only global resource center on FFS. The Platform also provides visibility to networks at national and regional levels, and facilitates exchanges between them. Through the Global FFS Platform, key products are being developed to support the quality of FFS programmes, such as the Global FFS Guidance Document. Building on the existing FFS networks, it helps identify key issues that emerge from the field. In particular, the Global Platform aims to collect, develop and analyze information relevant to the global community and integrate new ideas from previously unavailable documents. It is committed to facilitate exchanges between organizations, countries and institutions interested in community-based development and sustainable agriculture. Useful products can be designed to support the quality of FFS implementation in many FFS projects implemented by FAO (more than 70 ongoing projects just in Africa) and other institutions. Most projects with an FFS component collect data on FFS implementation as part of their monitoring and evaluation (M&E) system. However, the level of detail varies from programme to programme. Since each programme defines its documentation needs differently, there is great variation in the information collected. In some cases excellent and comprehensive documentation is available, while in other cases it is sketchy and inadequate. Very often, FFS projects put greater emphasis on field implementation than on documentation and lessonssharing. Therefore, there is a need to identify a harmonized and target-oriented set of core impact indicators relevant to FFS. Different partners are keen to know how well FFS programmes are implemented, and what changes transpired from them. Impact studies have been carried out in many places, sometimes by the projects themselves, other times by external investigators from universities or other institutions. Some studies have been published in peer-reviewed journals while many exist only as grey literature. In 2004, a study was done to synthesize existing impact studies on IPM-FFS (van den Berg, 2004). It showed that diverse methodologies, processes and tools were employed in these 1

7 studies. Particular challenges highlighted were capturing the diversity of impact areas (social, economic, environmental, policy) at various levels (trained individuals, households, communities, national institutions) while limiting costs and time for the studies. Furthermore, it proved difficult to provide an accurate picture of local realities, and to ensure that identified changes were indeed attributable to FFS interventions. The study revealed a demand for commonly agreed frameworks on basic data needed for FFS programmes, and for tools to capture innovations which can be refined based on local contexts. In 2014, another study reviewed the use of FFS for improving farming practices and farmer outcomes (Waddington et al., 2014) To update and improve data collection and impact studies of FFS, FAO is partnering with key organizations to develop a series of knowledge product to strengthen the process of monitoring, evaluation and learning of FFS programmes and activities. In 2018, two key products will be developed by FAO in collaboration with Wageningen University: (1) A Global Synthesis of existing evaluations of FFS programmes (covering the period); and (2) Strengthening the existing set of tools and providing a harmonized framework for improved impact assessment and M&E of FFS programmes. A critical step in this process has been to hold an international workshop with key FFS programme officers in order to collaborate in the development of the above-mentioned knowledge products. The workshop brought together Field School practitioners from different regions, backgrounds and organizations to work collectively on the preparation of a framework and toolbox for impact assessment and M&E of FFS programmes. 2

8 2. Workshop FAO and Thai Education Foundation organized the Global Workshop on Impact Assessment and Monitoring and Evaluation of Farmer Field School Programmes at the Sukosol Hotel in Bangkok, Thailand from 17 to 20 September The objectives of the workshop were to: 1. Review and validate the draft synthesis of evaluation studies prepared by Wageningen University. The researcher from Wageningen presented the methodology used, the structure and the main lessons learned which emerged, with implications for the setup of a harmonized M&E and impact assessment framework. This session aimed to give participants an opportunity to reflect on lessons learned and validate the draft synthesis; 2. Identify key challenges to effective impact assessment and MEL in FFS programmes. Participants brainstormed on the challenges they encountered and key challenges were e identified and validated by the group; 3. Develop a common impact assessment framework, including basic data needed for M&E and impact assessment of FFS programmes; and 4. Identify and share key tools for impact assessment and MEL of FFS, which can be refined based on context. The workshop was attended by 31 participants from different backgrounds and organizations in Asia (Bangladesh, Cambodia, India, Indonesia, Lao PDR, Myanmar, Nepal, Pakistan, Philippines, Thailand, Vietnam), Africa (Angola, Burkina Faso, Ethiopia, Kenya, Malawi, Mozambique, Rwanda), Near East (Egypt, Jordan) and South America (Brazil, Peru). The list of participants is given in Annex 1. In addition to the 31 participants another 4 were invited that were not able to join the workshop for different reasons beyond their control, representing different organizations and regions (FAO, government staff, CARE, from Africa (Kenya, Senegal) and Asia (China). The programme of the workshop is given in Annex 2. It started with a one-day field visit on Monday, 18 September and continued for three days at the hotel with group discussions, presentations, a poster session and a panel discussion. This report gives the day-by-day summary accounts of the various workshop activities. Monday, 18 September 2018 A field trip was organized on the first day to share and learn from Thai experience and share lessons-learned on monitoring and evaluation of FFS programmes at local level (see Annex 3 for more information on the field visits). The group visited an NFE-FFS 1 on rice agroecology at Chumchon Wat Tamlae Tong School and an NFE-FFS on pesticide risks at Ruamjai Prasit School in Prathumthani Province. In the afternoon, the workshop participants visited a group of farmers who had joined a Young Smart Farmers programme which encourages young professionals to take on a career in agriculture. The farmers explained their products, 1 A Farmer Field School under the Ministry of Education (Non Formal Education group) 3

9 marketing strategies and networking activities. All programmes visited where inspiring examples of government responses to two common problems: (1) Pesticide contamination at village level and contamination of food, and (2) a declining and aging rural population. The workshop participants identified the following success indicators for the visited programmes: Schools Increased awareness of pesticide risks to health and the environment Inclusion of agro-ecological issues in the school curriculum Integration of formal and non-formal education to develop future farmers Sustaining school food programmes Strong community support and collaboration Increased demand for clean food from parents and in market Linking FFS with public schools Attracting youth to employment in agriculture Young Smart Farmers Sustainable and adequate income Sacrifice their careers to find personal happiness and satisfaction Strong networking via internet and social media Diversification and market orientation of production Knowledge about marketing, value added and agricultural production Successful domestic and export marketing Reduction of climatic risks Access to credit and government support Applying different professional skills to agricultural entrepreneurship Attracting new young farmers into the program The following examples of M&E were observed in the visited programmes: Schools Monitoring pesticide impacts in the fields and the community Testing for pesticide residues in school lunches and in the urine of students Student s monitoring of pesticide storage in their parents homes Young Smart Farmers Use of modern technologies (drones) to monitor crop development Programme implementation: exchange visits and technical support Quality and yield of farm products Market demands through internet networks The field visits provided a good opportunity to learn more about the activities of the Thai Education Foundation and the use of FFS in public schools as an education tool. Thai authorities seem concerned about the aging rural population and that young people are not interested in agricultural and move to the cities. Both the Ministry of Education and the Ministry of Agriculture have initiated programmes to counter this trend and to ensure continued food self-sufficiency for the Thai population. The Thai experiences may be applicable to other parts 4

10 of the world and in the home countries of workshop participants. Furthermore, the visits provided practical examples of monitoring and evaluation activities at local programme levels. Tuesday, 19 September 2018 After a round of self-introductions, the workshop was formally opened with welcome addresses by Marut Jatiket for the Thai Education Foundation, Jan Willem Ketelaar for FAORAP and Marjon Fredrix for the workshop organizers. Following a brief feedback session about the field trip, the participants listed their expectations for the workshop. Besides expecting to learn more about how to conduct M&E and impact assessment, there was also a great interest in further developments of the FFS approach (see Annex 7 for more details). Poster session The participants shared and discussed 23 posters (see Annex 4 for pictures of all posters) which displayed information about their country s FFS projects and programmes, main entry points and topics, and the impacts achieved as the result of the FFS interventions. The posters exhibited a great variety of FFS activities. Summary of topics displayed in the posters: M&E Systems IA Results Crop production Animal husbandry Forest Soil and water (Bio)diversity Aquaculture Livelihood improvement Climate change IPM and pesticides Income Food security Nutrition, food safety Health, sanitation, family Knowledge Learning /experimentation Literacy Gender Relations/Women Employment Community Development Farmer networks Cooperation stakeholders Business management Group savings Marketing Quality Angola x x x x x x x x x Bangladesh x x x x x # # # x x x x Brazil x x x # x x # Burkina Faso x x x x x x x x Cambodia x x # x East Africa x x x x x x x Ethiopia x x x India x x x x Indonesia x x x x x Jordan x x x x x x x x Kenya x x x x Laos x x x x x Malawi x x x x x x x x x x Mozambique x x x x x x x x x Myanmar x x x x x # x x x x Nepal x x x x x x x x x x x Pakistan x x x x x x x x x Peru x x x x x Philippines x x x x x x x x Rwanda x x x x x x x x x Thailand x Vietnam x x x x x x x x x x 5

11 Yemen x x x x x = topic covered; # = detailed results presented The poster presentations showed that the FFS programmes aimed at achieving impact results in different domains such as increasing knowledge, attitudes and skills for food security and food safety, health issues, women empowerment and gender relations, strengthening of community groups and closer networking, as well as improving business and marketing skills. The different FFS dealt with crop production issues, animal husbandry, climate change and community development issues. Some of the posters contained specific impact results in terms of higher production efficiency, improved nutrition and marketing successes. A few countries had established elaborate systems of participatory self-monitoring for programme improvement and problem solving. Thus the poster session set the stage for discussing M&E in FFS programmes. Presentations To provide an introduction to the topics to be discussed during the workshop, two presentations were given (see Annex 5 for full presentations): Global FFS Review: Some thoughts on M&E by Henk van den Berg Monitoring and evaluation was explained as an essential part of learning in FFS projects; it draws conclusions and lessons learned for improving project implementation. Monitoring assesses quantitatively and qualitatively whether implementation was according to the work plan, while evaluation assesses the effects of knowledge, attitudes and skills as outcomes and impacts; which can also include unexpected effects. The way M&E is conducted depends whether we see FFS as a linear process of technology transfer where farmers adopt improved practices, or whether we see FFS as an educational investment for empowerment and continued learning. The latter is more complex as it looks at changes in mindset, adaptations, innovations, financial management and working together. 6

12 The objective of M&E is to improve the quality of our activities and steer them in such a way that they achieve the desired impacts. M&E analyzes and Causal stage Issue at hand reflects on our Does the programme make an Impacts impact? actions and is best done in a Are the activities making a participatory manner. Effects Outcomes difference? For doing so, only such data should be Outputs Are the activities effective? collected that is actually needed and Are the activities done right? used for evaluation. Interventions An important Are the right interventions done Design to address the problem? element of the M&E framework is the Problem identification Are pertinent problem and contributing factors addressed? causal chain (see graph) which helps us to understand how the activities lead to the outcomes; it can also help us to trace backwards to earlier inadequate steps when an outcome or impact was not fully achieved. Therefore all steps along the chain need to be monitored and evaluated. During FFS implementation, the introduction and strengthening of certain critical skills, social competencies, experimentation, confidence, group building or speaking out can lead to additional impact areas that have previously been overlooked. The M&E framework can be assembled in the form of a mandala or dart board (see graph). The mandala shows the causal I MP A CTS chains from the FFS interventions HUMA N S OCI A L OUTCOMES Quality of life, to the impacts in the personal Emancipation, empowerment group empowerment, Confidence, four domains of OUTP UTS Collective access to mindset, action, services/markets, human, social, good health Experimentation, networks, Trust, innovation, advocacy S K I L L S bonding, natural and exploration, Critical Social linkages negotiation financial capitals. In thinking skills an education FFS programme, all four Ecosystem Financial analysis skills domains are Practices Income, of ecosystem costs, profits, Savings, marketing important. tthe Food production, management loans, diversification, assets Financial peripheral impacts conservation (natural/physical) Resilience, security, of trickle down food security, poverty reduction, ecosystem services increased options effects on policy NA TURA L F I NA NCI A L makers, institutions and non-ffs farmers are missing in the representation. Activities Peripheral effects on policy, institutions and non-ffs farmers 7

13 The presenter concluded that the main challenges for M&E are to define and measure the indicators and to link causes and effects in a convincing manner. Following the presentation, the following issues were raised: Looking at an FFS as a group of farmers and not just as a school helps us focus on the social impacts; Some aspects in the mandala are compounded and cover multiple domains; thus, resilience not only applies to the natural domain, but is also affected by human, social and financial elements. Likewise, poverty not only relates to income but also to gender and empowerment; Extension systems have evolved from technology transfer to become innovation and education systems; thus FFS should not be positioned as an alternative to extension, but in support of a modern understanding of extension; Combining physical assets into the financial domain simplifies the mandala and is adequate for most FFS; however, the number of domains can be adjusted to the requirements of the project; Besides skills, attitudes and knowledge are also important direct outputs of FFS activities. Sometimes a skill can also be seen as a higher level outcome. More circles could be added to understand the causal chain, and labels are sometimes very theoretical and need to be better described; Many are confused about the meaning of outputs and outcomes. Outputs generally relate to the effectiveness of an activity, to farm-level or short-term effects, while outcomes assess whether they made a difference, or refer to community-level or long-term effects. All definitions have pros and cons; Impact studies also have to look at the quality of implementation; a lack of impact may be caused by a poor-quality implementation of FFS activities. Monitoring data should provide answers. Case Study of Asia s Impact Assessment of FFS Programmes on Pesticide Risk Reduction by Gerd Walter-Echols The project Pesticide Risk Reduction in South-East Asia started in 2007 with the objective of eliminating the use of hazardous and persistent agrochemicals through IPM farmer training. It aimed at better access and utilization of alternative pest management options and support for National Pest and Pesticide Management policy reform. One of the first activities was an impact assessment workshop which defined the impact targets (see table) and developed an impact assessment matrix. External research institutions were engaged in Cambodia and Vietnam to conduct two impact assessment studies. In Cambodia, a mixed vegetable and a dry rice production area were selected, while in Vietnam the studies were conducted on cabbage production in a suburban setting, and on winter melon production in a rural district. Based on matching village profiles and mappings of pesticide flow and exposure, study and control villages were selected. 8

14 The units of analysis were groups of farmers and villages. In Vietnam, community officials, pesticide dealers and a sample of pesticide users were interviewed. In Cambodia, FFS participants and non-ffs participants were sampled in the IPM villages. Corresponding data were collected from control villages without FFS-PRR interventions. Baseline data were collected in There were a total of 303 respondents in Cambodia and 364 in Vietnam. In Cambodia, the annual pesticide use per household, the practice of mixing different pesticides and the number of hours farmers were exposed to pesticides were some of the main risk indicators. In addition, the signs and symptoms of farmers feeling uncomfortable after spraying were collected for health effects. The Vietnam study included knowledge on pesticide risk factors and observations of practices. The impact data were collected in Results showed an increase in knowledge, a reduction in total and Class I pesticide use, less mixing of different pesticides, and improved disposal of pesticide containers and increased use of protective clothing. In Vietnam, community officials had a better knowledge and pesticide shop owners were more aware of regulations and improved their storage practices. All these factors have resulted in a reduction of pesticide poisonings and pesticide risks for applicators, consumers and the environment as measured with the Environmental Impact Quotient in Vietnam and a modified formula in Cambodia. 9

15 Effect on pesticide poisonings Cambodia % PRR-FFS Vegetable farmers PRR farmers 2015 % Exposed farmers PRR-FFS Rice farmers Control farmers An extension of the project made it possible to conduct a long-term impact study in 2015, seven years after the training of farmers in pesticide risk issues. However, at that time there were no longer control villages in the strict sense since many activities took place between 2010 and 2015 that influenced pesticide risks. Most importantly, Class I pesticides were banned in Cambodia in Furthermore, only 180 of the original respondents could be interviewed in Cambodia and 230 in Vietnam. The long term impact assessment results showed that many of the changes detected in 2010 were still found in Farmers had retained their knowledge of pesticide risk and were using less than 50% of the pesticides they used in The use of highly toxic pesticides was effectively eliminated because they had been banned in both countries. Many safety measures such as protective clothing were still used in This had a clear impact on the number of pesticide poisonings in Cambodia (see graph). It can be concluded that the pesticide risk situation has greatly improved in both countries. This had a clear impact on human health as measured by the dramatic reduction in reported pesticide poisoning signs and symptoms. It can be assumed that this further affected healthrelated expenditures, fewer sick days and a general increase in well-being. It was also shown in Cambodia that the number of natural enemies in the field were clearly higher in IPM villages. Thus, a reduction of environmental risks could also be observed Year Following the presentation, the following issues were raised: The control factor is a major issue in impact assessment studies. In addition to matching on village level, households also need to be matched, and the results need to be presented in the baseline; In the baseline, all groups of farmers should ideally be identical. However, this is difficult to achieve. Nevertheless, the relative changes between the groups indicate, which changes can be attributed to project activities. 1 st Group Session: Updating the M&E Mandala The workshop participants were divided into four groups which were then asked to populate the mandala with indicators at all levels in the four domains. 10

16 The result of this exercise gave a complex and somewhat confusing picture of the M&E mandala. In the following write-up, only the impact level indicators are presented. The complete list of all indicators at all levels is given in Annex 6. 11

17 Impact level indicators identified: Social Number of laws and community decisions passed in consultation with farmer groups Institutionalization of local policy Leadership and community development Percentage of male vs female in leadership positions in the community Increased number of social events in community, sense of belonging, mixing between social groups Quality of lives improves Connections between non-formal and formal education Natural Adaptation of best management practices Increased biodiversity Financial Increase in income Diversification of income sources Increase in profitability Human Food security Food safety and better nutrition Balance in role sharing Better access to information and communication technology During the discussion of the results, the following points were raised: Placing all indicators into a mandala creates a complex and confusing image; The mandala is useful for visualizing that an FFS can have impacts in multiple domains; This exercise identified a number of potential indicators which were not previously included such as negotiation skills, market access or financial tools; The mandala allocates the smallest space to the FFS interventions and the largest space to the impacts. This distorts the causal chains and gives the impression that a maximum of impact can be achieved with a minimum of inputs; The skills circle of the mandala should also include knowledge and attitudes; There are no clear distinctions between the domains and levels. This makes it sometimes difficult to place an indicator at a particular level in a specific domain; Many indicators are difficult to arrange in a causal chain, particularly in the social and human domains; Many indicators are too general and thus difficult to quantify and measure. The day finished with a dinner cruise on the Chao Phraya River. 12

18 Wednesday, 20 September nd Group Session: Causal Chain The groups were asked to clean up yesterday s results, select one impact as an example for showing the causal chain back to the skill level. The results in the different domains were as follows: Social Domain Group mobilization Analytical skills Awareness of norms Skills Decision making skills Presentation skills Technical skills Output Trust & accountability Responsibility Changes in behavior, attitudes and perception Outcome Increase in space of women in household Change in power and roles in households decision making Impact Empowerment Increase in leadership and community development Good cooperation in community and household Super impact Quality of lives Inclusive and equitable societies The cause-effect linkages and hierarchical order between the levels need to be more clearly described; The distinct causal paths from impacts to specific inputs need to be monitored. Human Domain Social skills: communication networking, leadership Technical skills No. of farmer-tofarmer training Improved cropping pattern Skills Financial skills: cost management No. of No. of Observation Analytical problem decisions skills skills analyses made Output No. of farmer No. of adopted good facilitators practices Outcome Access to % reduction of % increase in diversified food losses yield Impact Improved household food security in specific community No. of solved problems No. of farmers graduated No. of certified farms Indicators of human capital are professional competencies acquired through knowledge and skills training that allow an individual to work as a productive member of society; Possible indicators of human capital are critical thinking, creativity, innovation, ability to adapt, confidence, empowerment and health; The process of human capital development and the causal chain from FFS interventions to impacts seem difficult to describe. 13

19 Natural Domain AESA Agricultural knowledge, practice and technology Adoption of best management practices Area under conservation Skill Knowledge of environment and hazards Output Knowledge of agroclimatic systems Informed use of inputs Outcome Diversity of farming systems in landscape Impact Improved ecosystem services Knowledge of pests and natural enemies Access to agricultural information Diversification and intensification of production Productivity The skill level includes mostly knowledge items; The causal chain allows for the monitoring of measurable indicators: o Skills: Number of participants, topics, curriculum; o Outputs: Number of practices, quantity of inputs, number of crops; o Outcomes: Number of production systems, areas under farming systems; o Impact: Soil quality, percent organic matter, level of soil nitrogen, water, carbon, biodiversity, pollinators, natural enemies. Financial Domain Skill Negotiation skills Cost-benefit analysis Output Innovation / Improved production Market research re Group promotion premium quality in quality and quantity demand Outcome Response to market demand Work with market agents Impact Increased income Sustainable market Final Impact Well being The described causal chain is target-oriented and logical; The financial domain links closely with the human, social and natural domains. Overall observations: Generally, multiple skills are needed to achieve a specific impact. These interactions cannot be easily and clearly represented in the mandala; The hierarchical order and causal relationship from one level to the next seems sometimes unclear; The process of how a specific intervention leads to the development of a certain skill, and how this skill has to be applied to produce the desired results is not clearly described; When clear impact targets were formulated (e.g. increased income), a clear causal chain can be established and the backward path to the interventions can be described in a target-oriented manner; 14

20 For unexpected impacts, only the process of FFS implementation and the possible outcomes can be described because the causal chain is unclear; The institutional dimension is difficult to attribute to a specific domain; All domains need to be strengthened so that the voiceless can become relevant in their communities. 3 rd Group Session: Indicators and Methods of verification The groups were asked to list the methods they would use to collect data on these indicators. What are the advantages and disadvantages of the methods? Level: Activities and inputs Description Indicator Method of verification FFS farmers/group graduated Farmer capacities for agro-ecosystem management No. of farmers graduated as per graduation certification criteria No. of action plans prepared by graduated farmers No. of post-ffs activities established No. of participants No. of thematic areas No. of curriculums FFS record book (ODK/ e- FFS) Record on action plan Open discussion Focus Group Discussion FFS records Level: Skill Description Indicator Method of verification Awareness of norms Proportion of participants taking steps towards common norms Observation Focus Group Discussion Presentation skills Decision making Observation skill Negotiation skills Cost-benefit analysis Group mobilization No. of times having presented (different FFS participants) No. of participants contributing to key decisions made No. of AESA conducted No. of analyses conducted by farmers No. of observation of pest/ predators No. of decisions made by farmers Decision pattern No. of contracts finalized No. of analyses completed No. of commodity identified ($) Group norms - roles and responsibilities - contributions - minute of meeting Key Informant Interview FFS records Observation Interview FFS record book, observation, Semi Structured Interview Group records Observation 15

21 Level: Output Description Indicator Method of verification Change in behavior and attitude Proportion of participants taking steps toward changing behavior Observation Increase in space of Level of satisfaction among women in Focus Group Discussion women in household decision making household decision making Key Informant Interview Household interview Balanced production systems Improved production Market research Product promotion Yield in kg/ha No. farming systems Areas cultivated Reduced cost of production Increased efficiency Increased yield Increase % premium grade products Increased profit (often measured in % increase) Volume demand identified No. potential buyers No. of buyers contacted No. fairs or market places attended PRA Records Survey Records Interviews Research Reporting Level: Outcome Description Indicator Method of verification Trust Responsibility Accountability No. of conflicts (Issue: Accuracy of group records) Observations Interview with members Group records Farmer adopted /adapted good practices Response to Market demand Sustainable Market: Premium quality in response to market demand No. of farmers adopted/adapted practices in their farm No. of practices disseminated and adapted by FFS farmers Timing of production Percentage of premium product Certification of premium product Percent production increase of demanded product Focus Group Discussion Individual in-depth interviews Key Informant Interview Field visit, record Level: Impact Description Indicator Method of verification Good cooperation in community and household Level of satisfaction Interview PRA Leadership and community development Inclusive and equitable societies Improved human capability Strengthened production capacity Sustainability of farm and landscape Improved livelihood No. of women in leadership position Level of satisfaction of community about their involvement in society development Yield No. of crops Extent of decision making by FFS farmers % soil organic matter Water quality Population densities Farmers are happy to stay in their profession Interview with village leaders/local authority and women members Interview, PRA Social mapping Record, survey Focus group discussions Individual in-depth interviews Lab analysis Field observation Only one group elaborated on the advantages and limitations of certain methods. 16

22 Comments and observations: The presented group results are a step toward an M&E framework and toolkit for FFS; measurable Indicators provide a framework for monitoring; The examples show that often different and complementary causal chains need to be initiated in order to achieve one specific outcome. Such a cause-effect network is difficult to display in a mandala and may be better depicted in a flow chart; The format used resembles an upside-down logframe: Instead of starting with the skills level, a logframe normally starts with the project s purpose and goal, and then lists the necessary results, outputs and activities which are needed for target achievement. In addition, it lists the success indicators, means of verification and assumptions; Natural and human capital is sometimes difficult to differentiate; e.g. AESA is a format for recording observations in different domains; The skills layer in the mandala is misleading. An FFS provides for knowledge acquisition, skills training and attitudes reflections. They are all equally important in an educational approach. Thus the direct outputs of FFS interventions are increased knowledge, skill competency and changed attitudes, and not just skills ; On the output level, the M&E framework needs to include the understanding of basic concepts such as damage compensation, predator-prey relationships, supply and demand market regulation, value chain functions, ecological and financial resilience; These are indicators of empowerment. Sometimes, the understanding of basic concepts is a prerequisite for acquiring additional skills. These complex causal chains may require additional layers and descriptions in an M&E framework; Business development and entrepreneurship are important outcomes in some FFS; Some outcomes may be unexpected because the causal chains are poorly understood or overlooked in the planning process; Other outcomes are unexpected because it is difficult to predict how people will use their educational gains; Creating trust between conflicting groups can be the explicit goal of FFS or an unexpected impact; peace can be an important FFS impact goal; Some social impacts (gender, domestic violence) are not exclusive to FFS and may also be covered by other community organizations or NGOs; If farmers increase and improve their production but cannot sell it, no positive effects or impacts have been achieved; Social taboos can prevent the adoption of a skill, e.g. compost preparation; National policies can be crucial and critical for FFS success and therefore need to be included in an M&E framework; Organizational impacts may be part of a conducive environment which needs to be created through activities and monitored; The institutionalization of facilitator training in learning institutions is an important indicator for the long-term success of FFS and needs a place in the M&E framework; Market institutions are an important part of the financial capital and part of a conducive environment; Impacts on institutions, public extension system or cooperatives need additional indicators when only selected FFS elements are adopted. 17

23 Panel Discussion Question 1: If we take a bird eye view, what has changed in and around FFS since it started? What is most likely to change in the future? Kevin Gallagher First FFS in 1988 were in response to pest outbreaks in rice and aimed at solving a pesticide problem. Since then the world has changed. It was realized that FFS created organizations on village level, and that has led to a human dimension. Today we look at social, human, technical and natural aspects similar to the Sustainable Development Goals. However, many of our ministries are still more concerned about pests than farmer because their mandate is to produce food. Thus, while FFS have changed, the conditions in which they are implemented have not changed so much. Deborah Duveskog First, FFSs were like a clubs with a common approach, common beliefs and a fixed method. Over the years, this club feeling had to open up to a bigger perspective of agricultural and community development. This has strengthened the FFS approach. Now, FFS has to fit into a bigger picture of innovation system or extension, combining different aspects such as savings & credit, community disaster risk reduction, watershed management or diffusion through farmer promotors. From originally only serving groups, FFS now reaches masses and whole communities. This has sometimes compromised quality. Sandar Myo In Myanmar, FFS started in conflict areas where the government extension service could not reach. After that, it became attractive to the donor community for food security, poverty 18

24 reduction, education and livelihood improvement. Despite the success of FFS, it was found that farmers are becoming poorer. Many things are changing such as consumer behavior and farmer lifestyle. After FFS, farmers are interested in markets. Lately, value chains have become the focus of donors. Now, market-linked FFS are changing the style and approach of FFS and there is more focus on food safety and quality in the context of economic integration and cross-border trade. FFS now addresses these market requirements. Max Ochoa The FFS methodology is still attractive for development programmes. However, FFS had to adapt to new challenges and new actors. The attractiveness of FFS has also created risks as the pesticide industry used them to promote their products, or governments promoted their political agenda. Currently, the new challenges are climate change, increasing food production and business skills. This is a good moment to adapt the FFS methodology to these challenges. Therefore, we need to analyze the past impacts in the communities in order to improve the methodology. Question 2: Despite many examples of impacts, the agricultural situation is still precarious as inputs rise and new pest outbreaks happen. If we see the impacts at a project level, but not at a broader level, what are we missing? What does this mean for our strategies? What should we do differently? Kevin Gallagher We are seeing a move away from ministries as government extension services are declining. Many FFS are now led by the markets. In Sierra Leone, FFS helped cacao farmers get into fair trade, and in Mongolia a participatory guarantee scheme for organic goods is combined with technical FFS. With the rise of the private sector we will see an increasing need outside the governments. Therefore we need to sharpen our view how FFS and private sector fit together to match their needs. Deborah Duveskog The current institutional environment is making it harder for FFS. There has been a shift from the environmental movement and participatory approaches to a more technical and inputdriven new green revolution. This makes it harder to advocate FFS. Market forces are strong and decisions are often political and serving other interests. To promote FFS, we need to be clever and sometimes manipulative. Sandar Myo Most FFS practitioners are very much project-oriented. But often we see the impact only after the end of a project. We need to respond more to the immediate needs of farmers. We are missing the market linkage and helping farmers to get collective access to the market. However, sometimes only few farmers make the effort to get certified and we have to be patient to see the impact of our efforts. 19

25 Max Ochoa We need to learn more about our experiences and promote good methodologies in other countries. We need to show decision makers the importance of our methodologies and how they can be part of development strategies. We also have more opportunities in the relation with the private and public sectors. In coffee, for example, most of the funding is coming from the private sector. The International Coffee Organization with contacts to governments, consumers and NGOs is a platform that can promote this process on a large scale. There are more opportunities to convince donors to promote this process in other commodities such as cacao and cotton to support the farmers. Question 3: We have experienced that institutionalization is not just a linear process from pilot activities to government policies. Where have you seen this institutionalization make a difference? Maybe institutionalization is not just about governments, but also about farmer organizations. What you are your views on Institutionalization? What does it mean, and what do we do with that? Max Ochoa We need to reflect on turning the FFS methodology into strategies. Maybe we should not focus on convincing government officers who are then transferred to other positions. Institutionalization is not just for the public sector. We need to create strategies to have a longterm interventions. Sandar Myo Myanmar is trying to institutionalize the support by ethical and responsible private sector companies of the FFS demonstration fields. There are also FFS programmes that do not use the term FFS. Kevin Gallagher There are lots of initiatives that do not have pilot stages but use FFS because it makes sense. A hands-on approach is probably the best way for the private sector. Deborah Duveskog Extension has been in a crisis and a lot of donors have pulled out. In this situation, a linear process does not happen and we need a parallel pathway. It may therefore be already positive to find entry points to embed elements of FFS such as group-focus or problem-based in response to farmer s demands. It is not a take-it or leave-it situation. In the long run we can create a right platform for the field school approach. Anne-Sophie Poisot (facilitator of the panel session) Maybe we should not simply look at institutionalizing FFS, but on institutionalizing field school values such as empowerment and ecosystem approaches. This may be more valuable than a mechanical application of the FFS structure without having understood the FFS values and principles. 20

26 Question 4: Why and ultimately for whom do we measure impact? Kevin Gallagher Ministers may be interested in the FFS contributions to the SDGs. But ultimately, we have to show the impact to farmers, and in that case, the market is our success indicator. Actually, many of the indicators are for farmers and not for anyone else. However, as it has happened in one case, an external evaluator reminded the project that the most important matter they had created was trust which then led to economic development. Self-assessment tools are really important and records are kept as much for oneself as for an outside monitoring. Deborah Duveskog Most of the monitoring should be for the purpose of creating a learning organization, maybe at community level or project level. The impact assessment we do mostly for the donor. Often these studies have not generated information which the people involved did not already know. Thus impact assessment is sometimes the cost for staying in the development business. Max Ochoa We need to create a new culture for project implementation. Many donors do not want to invest in administrative expenses, including M&E. They want to understand the process, but do not want to provide funds. However, these types of assessments are expensive. We need strategies to develop capacities and to document the process. We need to be creative about using new technologies to collect data and disseminate our findings. Sandar Myo M&E creates ways to see the impact of FFS programmes in areas other than IPM. It can be incorporated into private sector activities for boosting quality and creating sustainable markets. Anne-Sophie Poisot It appears that different stakeholders need different tools and types of data. A donor may need sophisticated controlled studies, while a policy maker may already be convinced after talking to a group of farmers. To collect the information, we have a variety of tools we can use. 21

27 4 th Group Session: M&E tools for project management, FFS quality and learning The groups were instructed to answer the following questions: What data are you currently collecting? How are they collected? What developments are there? What are additional tools to collect data? What processes are in place and what tools are used to assure quality of FFS? What mechanisms can you use to correct/ improve? How to systematize lessons learned during implementation? What Who How FFS Planning Information Baseline participants/farms/ landscape situation Farmer group Facilitators Survey/questionnaire Focus group discussion Social data - Population indexes - Problems - Qualifications Infrastructure - Roads - Transport - Irrigation M&E design at project planning during FFS implementation FFS Implementation Monitoring Facilitators - profile Group/Village - Gender - Village profile - Host team profile - Farmer profile - Attendance - Leadership Agronomy - Pest and diseases - Natural enemies - Field condition Meteorological data - Rainfall - Humidity - Temperature Cropping data - Types of crops Equipment FFS Learning - FFS plot - AESA - Experiments - Field days - Technical topics covered Capacity of evaluators (=weakness) Who will summarize data? Facilitator Extension worker Farmer Village leaders Members, etc. Farmer group Facilitators Technicians Farmer, facilitator Farmer Field Book PRA Questionnaire Records Records (during and after) FFS record book/registry: participants, FFS, field Online: Open Data Kit, e-ffs (before, during FFS) AESA Meteorological station AESA, observations (weekly/fortnightly/monthly) Hand book 22

28 What Who How FFS Process and Impact Monitoring Knowledge, attitudes and practices of farmers before FFS Process quality Impacts - indicators Food security - Diversity - Meals - Hunger gap Economic analysis - Productivity - Labor Stories of changes - Comments on changes Farmers Facilitators Supervisors Third parties Pre-test, PRA, case study (before FFS) Survey/questionnaire Focus group discussion Farmer Field Book Pre-test Survey, studies, data box, participation discussion, FFS Focus group discussion Questionnaire Additional tools IT / mobile phones - Financial data app - Farm economic app - Real time data app - Geotagging information - Tools for documentation Plant clinics Enabling environment Organizational information Feedback evaluation Quality assurance processes Incorporate M&E strategies/ tools in ToT Planning/evaluation meeting with multistakeholders: farmers, facilitators Workshop for trainers to share experiences Feedback sessions for improvements Follow-up Meetings to different level: local, national, regional, international Comments and observations: Data collection should be based on indicators to avoid that resources are wasted on collecting redundant data that will never be used; Color schemes for AESA evaluations are suitable for illiterate farmers; New ICT technologies can reduce the time for record keeping; most farmers have mobile phones; Instead of collecting FFS record books at the end of the season, pictures of the records can be uploaded continuously; A person is needed to compile and summarize the data collected by facilitators and farmers; Elaborate impact assessments are costly and therefore often skipped in projects; The collection methods could be harmonized so that the data produced are comparable between programmes; The types of M&E data collected should be tailor made towards the targeted impacts of each programme; The purpose of M&E is for learning and programme improvement; The day closed with two evaluation exercises (presented in Annex 7): 23

29 1. Write one key learning insight of the day; 2. Give questions or requests for clarification. 24

30 Thursday, 21 September 2018 Plenary Session The main points of discussions during the earlier days were summarized and commented upon by Henk and Gerd, and further discussed during the session. Review of Key Learning items Impact chain analysis using mandala How to formulate FFS indicators for impact FFS programs need to improve M&E to measure impact Identifying indicators and method of data collection for FFS Self-assessment tools are critical We have to think about ICT tools in FFS M&E Need for additional tools for improving FFS follow-up of FFS e.g. ICT Institutionalization of FFS is not an end in itself FFS programs need to adapt to emerging needs and varying environments FFS don t create markets for farmers but give farmers the means to understand the dynamics Review of impact targets Human domain o Confidence through better understanding and knowledge o Generation of new knowledge through experimentation o Capacities to generate income, support family and manage farm o Food security o Health, food safety and occupational safety Social domain o Increased trust and cooperation at community and household level o Inclusive and equitable society (no gender or ethnic discrimination) o Leadership abilities o Collective self-help capacities Natural domain o Sustainable ecosystem services, fewer pest outbreaks o Productive cropping system and landscape Financial domain o Access to sustainable markets o Increased income and reduced financial risks on investment Comments: The social domain may be enlarged into a social-political domain to include targets for institutionalization, political advocacy and an enabling environment. The institutionalization of facilitation skills and participatory methods in universities and agricultural training institutes is an important goal. 25

31 General goals such as empowerment, quality or improvement need to be defined in order to become operational. Review of Impact Assessment (an overview presented by Gerd Walter Echolls) Impact assessment begins with the definition of impact targets and the formulation of indicators for their successful achievement (see graph). It is useful to arrange all information in an impact assessment framework or matrix which includes the causal chains from the project interventions to the targets. The matrix provides a framework for reviewing the FFS curriculum to ensure that all target areas are adequately covered in the training sessions with sufficient time and learning exercises. Step 4 Participatory M&E Self- Monitoring Joint Reflection Joint Decisions Impact Assessment Steps Step 1 Target Setting Step 2 Definition of Indicators Step 3 Curriculum Review Step 5a Baseline Data Collection Step 5b Impact Data Collection Step 5c Long-Term Impact Data The impact assessment matrix presents a framework for monitoring the activities and processes towards target achievement. It provides a checklist for all information needed to document the causal chains as well as the time and method for data collection. Some information may be collected as part of the project s routine M&E and self-assessment activities; others may require special studies such as field observations, case studies or farm-household surveys. A survey questionnaire should cover some key information about the respondents (for matching purposes) and information about the factors that are targeted for change. For assessing the impact, the samples and time frame need to be defined. An impact study always compares a representative sample of FFS farmers with a matching sample of control farmers from villages in which no FFS has been conducted. A representative sample should take into account the total population in relation to the variable of interest, the precision level of the result, and the heterogeneity of the population in relation to the variable of interest (intra class correlation). The samples should be taken from at least three villages with more than 10 respondents per village. To document the changes, one set of data needs to be collected before the project intervention and another one afterwards when farmers no longer receive guidance from the project. An impact assessment study may include multiple comparisons such as before vs. after, FFS village vs. control village, or FFS farmer vs. non-ffs farmer (exposed) in the same village. When an impact target is clearly defined, often little information is needed to verify its change. However, when an impact target is fuzzy, then a broad range of information is collected with 26

32 the hope that something useful is included. Thus the volume, time and effort of impact assessment largely depend on the quality of planning. Comments: Assessing the impact of education is complex and often only visible after many years; Impact studies may require a combination of different investigations: for some topics rough data are adequate, for others rigorous studies are required; Information that can be collected through monitoring and self-assessments should not be included in expensive farm-household surveys. Information required for project steering should be collected through participatory M&E activities; The impacts investigated in specific impact studies must be of a high priority and relevant to the project s objectives; Impact assessments should examine the effects of outcomes such as pesticide reduction, income increases or knowledge gains on the lives of rural households; A qualitative case study needs a quantitative dimension which states how many farmers or which portion of the population it describes, i.e. a clear description of the methodology used; The timeframe of impact studies depends on the objectives. Some effects can be measured immediately while others can only be found after several years; Impact assessments need to describe the context in which the interventions were carried out; for example, women empowerment has a different meaning in different cultures; A sample must be representative of a well-defined larger population. A sample is credible when it is shown that it matches in the key characteristics; Each type of information needs be collected using a suitable method; Farmers are often aware of the obvious changes in the learning process and empowerment. A quick count of how many farmers experienced a certain change can provide both quantitative and qualitative impact information; The impact of new technologies can only be assessed after it has been adopted by a sizable number of farmers. Sometimes this may take many years; FFS impact is not about adoption, but about adaptation and making the best decision for the local situation. While the decisions may be different, the results should have common characteristics such as maximum profit, a minimum of pesticides and external inputs and a healthy agro-ecosystem as defined by appropriate indicators; Control groups are difficult to find and they cannot be controlled. Therefore it is important to monitor the context during programme implementation; Data collection should follow the principle as little as possible and as much as necessary, which has to be decided on a case by case basis; Good income data are sometimes difficult to get. They can be verified through indirect calculations of partial budgets and gross margins; Non-FFS farmers in the FFS villages (exposed farmers) show the diffusion effect of FFS. However, only knowledge, attitudes and simple practices can diffuse, while skills always have to be individually acquired through practice under expert guidance; It is important to send results back to the local level for verification. 27

33 Presentations Experience of International Potato Center (CIP) with Impact Assessment by Willy Pradel Caceres In general, impact assessment has two dimensions: ex-ante and ex-post; for this workshop, we focus on ex-post impact assessment. Closely related to ex-post impact assessment is the concept of up- and out-scaling, or how many people one reaches with the intervention. For upscaling, it is important to know whom one can reach and what changes one can produce. In general, CIP interventions focus on positive changes in productivity, but also nutrition and income. For impact studies on FFS, it separated three groups: (1) FFS participants; (2) participants in projects related to IPM; and (3) not related to any training. The impact studies conducted found positive changes in knowledge. We then analyzed the effect of knowledge on productivity. This type of impact assessment is very time consuming because one needs comparable groups. Through propensity score matching, one can select members of the intervention group that are comparable to members of the control group in all factors except one: FFS participation. When doing impact assessment studies, one has to be very specific on the research question and separate it from all other impacts that might be occurring. Now, there is a trend away from this type of quantitative studies. For example, in Asia, FFS have evolved into farmer business schools (FBS), and changes tend to be income-related. FBS are included in IFAD investment projects that aim at improving the people s well-being through infrastructure investments. Those projects need to show how income affects farm household livelihoods. For that, the Most Significant Change method was used whereby in addition to the impacts on productivity and income, stories of the most significant changes were collected through participatory M&E methods and which then became part of the impact assessment report. These stories narrate the causal process from the project interventions to the big changes in well-being of the people. Thus in an impact assessment study one should have both rigorous impact numbers and credible stories about the impact and the drivers of change. Impact assessment always involves the comparison of a treatment group sample and a control group sample. There are different ways of selecting control groups, and sometimes assumptions must be applied. However, villages and farmers must be matched, i.e. they must be similar to each other. The final impact assessment reports must state clearly what were the objectives of the study, what the variables were, what the context was, and which farmers with which characteristics changed in what way. The most common impact studies look at economic impacts. When participatory methods are added, it is much easier to link the financial impacts to social and human aspects. Comments: Sometimes internal participatory evaluations are not credible because in a group interview people are shy to talk about their income. This can be overcome with an independent external evaluator. On the other hand, if impact evaluation is for learning and improvement, it must be done internally in order to create powerful insights among the participants. In Africa, intercropping is normal and farmers have income from many crops. To assess the impact, one needs to apply a system view and assess the whole farm. An effect can be highly significant on a particular crop, but insignificant on farm level. 28

34 External evaluators often have limited time and focus on just a few specific variables and thus may miss the bigger picture and some unexpected changes. It is therefore good to have both internal and external evaluations. Participatory methods are best for an internal evaluation, and quantitative methods for an outside one. The selection of an objective critical evaluator is essential, but this is often compromised in projects when good results want to be shown. Also, projects need to prioritize what they want to evaluate, e.g. whether a pest problem has been solved or whether income has increased. Global FFS Review: Overview of Findings by Henk van den Berg The Global FFS Review consists of (1) case studies from Malawi and Indonesia, (2) a global survey of the status of FFS implementation, and (3) a review of impact studies that have been documented. Only the results of the case studies were presented. There is a lot of information in FFS that might be missed by impact studies, particularly in the human and social domains. Two case studies were selected to examine whether FFS are still relevant: Do they help farmers solve their actual problems? Does the FFS interact in its institutional and political environment? How does the FFS contribute to rural development? From Africa, Malawi was selected. It is mainly maize-based rainfed subsistence farming facing a number of climatic and natural challenges. FFS started in 1995 with IPM, but it was not well accepted. Since 2010, there were a number of FFS initiatives focusing on climate change adaptation, market-oriented farming and diversification of income sources. Extension services have to implement the government subsidy programme while trying to be more bottom-up problem oriented and demand driven. FFS implementation is decentralized by the district offices and quite well accepted. It has developed into a platform for integrating different agricultural projects such as on nutrition, irrigation, composting, marketing or savings & loans into the FFS topics. This fostered a holistic view on the livelihood situation. Most FFS are farmer operated and continue for many years with an ongoing process of learning and adaptation, and farmers have diversified their sources of income. In Indonesia, FFS was introduced in 1988 as a response to pest outbreaks in rice, and it developed in a huge national IPM programme. However, funding stopped in 2002 and there were no major donors for the past 16 years. The only custodians of FFS were the trained alumni farmers, some local agencies and NGOs. While Indonesian farmers have become richer, there are severe challenges such as availability of water and land, environmental degradation and new pest outbreaks. The institutional memory of the IPM-FFS programme is fading. The ban of broad-spectrum pesticides from 1986 has been surpassed by new products and formulations. Extension activities have become increasingly technical and less environmental. Nevertheless, IPM-FFS is still promoted by the Plant Protection Directorate, but there is little interaction with other agencies such as the Extension Department. However, the trained FFS facilitators are retiring and the new officers are untrained. The Indonesian IPM Farmer Association with currently 1.2 million members has grown over the years to promote community-based IPM and farmers rights. The Field Foundation (Indonesian NGO with strong roots in the FFS IPM programme) has shown how the FFS approach can be used to address broader farmer issues such as community forest, water distribution, biogas production or 29

35 wastewater management. Taking the FFS model to new areas has created new models of innovation. These examples showed that FFS gave farmers to opportunity to improve their livelihoods and are still relevant. The continued activities of alumni groups indicate that empowerment took place. Advocacy could be improved by selecting a more pluralistic cadre of FFS facilitators that could interact and create a broad support. Thus case studies are able to describe a bigger picture and are a tool to supplement M&E and impact assessment studies. Comments: AESA is primarily a tool for recording group observations which can be applied to any other experiential learning situation. The tool got its name from IPM-FFS where its focus was on agro-ecosystems. The name FFS triggers different reactions with different people. Important is the concept, the underlying principles and the community outreach. Sometimes groups change to a different name after graduation. Successful community outreach aims at being farmer driven and leaving nobody behind. FFS must not be crop-based, but must be farmer-centered. The Global Review might also include FFS processes that do not carry the name FFS. FFS Impact Meta-Study Eastern Africa by Crispin Mwatate This study was conducted by a 4-member research team. Its objective was to systematically collect and synthesize evidence on the effectiveness of FFS, based on 29 relevant documents produced in the 10 EGAD member states in since The major technical focuses of the FFS were on food crops, soil fertility, land management and conservation. The objectives were predominantly empowerment and technology dissemination. The major factors contributing to sustainability were support from the government, establishment of diversified income-generating activities and savings & credit schemes. Farmers felt empowered being given a chance to share experiences, test new technologies, and express themselves in group sessions. They also developed their own impact indicators. Where participants were involved in problem identification and decision making, there was sustainability and tangible impact. It was clear that facilitator training and competence was a critical element of a successful FFS programme. Non-technology factors included market access, favourable prices, climate and soil fertility. Counterproductive factors were participant selection due to political and other factors, lack of inclusiveness, lack of relevance of introduced technologies and lack of clear M&E indicators and targets. Programs could be made stronger through a joint training of master trainers and anchoring FFS in the curricula of teaching institutions. Furthermore, there was a need to create more awareness about the FFS approach among all stakeholders, institutionalize FFS in the extension policy and create networks among FFS practitioners. Comments: The list of all documents that were sampled for this study can be provided. 30

36 5 th Group Session: How to strengthen M&E? The participants were asked to answer two questions: How do you strengthen M&E in your programme and what are the requirements for strengthening and mainstreaming M&E? Question 1: How do you strengthen M&E in your programme? The following responses were given: M&E framework Design an appropriate M&E framework Define the scope of M&E Define roles and responsibilities of different stakeholders Regularly review and update M&E system Tools Develop manual with clear protocols to follow: design, sampling, time, frequency Develop customized tools/methods for data collection Develop work plan and checklists for data collection Use harmonized data collection formats Make better use of smartphones and applications for monitoring and uploading data Organize and conduct peer monitoring Evaluate effects at different levels Resources Budget for M&E Dedicated staff and work space for M&E Build M&E capacity for project team and extension workers Establish a data management system Establish coordination committee for M&E Feedback Establish platform for regular feedback and learning Communicate results to communities and stakeholders at district and national levels Link to relevant institutions and national data centers Comments: Distinguish between activity monitoring, progress monitoring and impact monitoring. Quality indicators and quality matrixes are important tools for monitoring project implementation. They are needed for impact assessment to exclude bad performance as a factor that affected the results. Some programmes have assigned M&E to dedicated teams while others have incorporated M&E into the regular learning process. Peer monitoring with different government departments can be a way to acquaint them with FFS. Participatory M&E is a requirement for learning by FFS groups, facilitators and communities. All actors need to practice critical thinking to fix problems and improve their performance. 31

37 Participatory self-assessments show what is happening in the communities. Taking pictures of changes can be a tool for self-reflection and advocacy. Some data are only relevant at the collection level and do not need to be passed on; e.g. weekly AESA is for M&E of the FFS field. However, AESA results may be summarized in a matrix at the end of the season and shared with other groups. Only some information of the FFS record needs to be collated: e.g. the attendance record can be condensed into the number of farmers who have attended a minimum number of sessions. The information forwarded to district and national level is for project steering and critical analysis. The time of data collection needs defined in work plan; baseline data need to be collected before FFS interventions. M&E data should be regularly checked whether they were used for critical analysis and decision making; unused data can be excluded from future collection. M&E findings may lead to adjustments of the project logframe during programme reviews. Question 2: What are the requirements for strengthening and mainstreaming M&E? The following responses were given: Factor potential indicators into project framework during programme formulation; Clarify relevance of each indicator in project framework; Establish work plan and time frame, incl. for end of project situation; Build requisite capacity for importance of proper data collection; Ensure that feedback loops are in place to different stakeholders. 6 th Group Session: Action Plans The workshop participants were asked to brainstorm about actions points for follow-up after this workshop. The following responses were given: Workshop results Elaborate mandala Share presentations and reports to workshop participants Distribute synthesis of workshop and discuss with FFS community to get additional input M&E toolkit Share/fit participatory M&E tool kit Develop/revise participatory M &E framework (if there is a need) Draft toolkit circulated for our input Expedite sharing the toolbox to assist with national level briefings 32

38 National follow-up workshop Identify and apply for HQ funds for workshop Organize follow-up workshop/learning events Share findings/outcome of the workshop with M&E colleagues/partners National workshops on the M&E (FAO funds) M&E training Incorporate an M&E module into the training curricula and train master trainers and facilitators in M&E tools Networking and sharing Commit to collecting M&E data and materials for sharing Use FFS platform for sharing FFS results from the field Explore the use of M&E applications for basic information collection Establish and strengthen e-network; use /WhatsApp to connect M&E staff Closing of workshop A review of the expectations to the workshop (see Annex 7) showed that most of the expectations were accomplished. Some were only partially fulfilled, and especially the toolkit still needs further attention. The workshop was closed with short speeches by Marjon, Jan Willem Ketelaar and Marut Jatiket. They thanked the organizers and the ThaiEducation team for the workshop preparation and implementation, the participants for their active and productive contributions and Dada for the workshop facilitation. Special thanks went to Kevin Gallagher for his efforts to join the workshop. 33

39 3. Conclusion The workshop brought together 31 Field School practitioners from different backgrounds and organizations in Asia, Africa, Near East and South America, to work collectively on the elaboration of a common framework and toolbox for impact assessment and monitoring, evaluation and learning of FFS programmes. During a field trip, participants learned from the Thai experience on monitoring and evaluation of non-formal FFS programmes at local elementary school level, as well as from a group of Young Smart Farmers who explained their products, marketing and networking. The M&E experiences of the workshop participants in their respective countries were shared and discussed during a poster session with 23 exhibits. A panel discussion elaborated on changes in the FFS approach since its inception, new challenges for practitioners, as well as issues of institutionalization and impact assessment. Furthermore, presentations were given to introduce the concept and methods for M&E and to report on a case study on the impact of pesticide risk reduction in Asia. A metaevaluation study in Eastern Africa confirmed field schools as an effective extension approach. Moreover, the preliminary findings from a global FFS review study were presented which examined the spectrum of impacts and the conditions and causes of impacts. The main work of the workshop, however, took place in groups where the participants elaborated on a common framework for Impact Assessment and M&E. They (1) first defined additional impact indicators in the human, social, financial and natural domain and then (2) selected one impact target and developed the causal chain back to the skills level. Next, (3) the groups defined the indicators at the different levels and (4) the corresponding methods for who, what and how of M&E data collection. Finally (5), they brainstormed on the measures necessary to strengthen M&E in FFS programmes and (6) listed the next steps to be undertaken. This harmonized impact assessment framework allows for the identification of M&E and impact data in order to effectively communicate quantitative and qualitative impacts of FFS programmes. The results of the workshop will flow directly into the global synthesis of existing evaluations of FFS programmes and into a set of tools to be provided in support of strengthening impact assessment and M&E of FFS programmes. 34

40 Annexes Annex 1: List of Participants... Annex 2: Workshop Programme... Annex 3: Field Trip... Annex 4: Posters... Annex 5: Presentations... Annex 6: Group Sessions... Annex 7: Expectations, Feedback and Final Evaluation

41 Annex 1 - LIST OF PARTICIPANTS No. Country Name Organization Address 1 Angola Mr. Cyprien Ndambi Ndoki FAO Ndoki.Ndambi@fao.org 2 Bangladesh Mr. Syed Md Rafiqul Amin DAE natne@ifmcbd.org 3 Brazil Mr. Maximo Gerardo Ochoa Jacome Hans Neumann max.ochoa@hrnstiftung.org Stiftung 4 Burkina Faso Mr. Tiko Hema FAO Tiko.Hema@fao.org 5 Cambodia Mr. Chou Cheythyrith Government thyrith72@gmail.com 6 Egypt Mr. Alfredo Impiglia FAO/RNE Alfredo.Impiglia@fao.org 7 Ethiopia Mr. Aresawum Mengesha TemTim FAO Aresawum.Mengesha@fao.org 8 Germany Mr. Hendrik van den Berg Wageningen University vandenberg.henk@gmail.com 9 Germany Mr. Gerd Walter-Echols Ex-FAO gerd.walterechols@gmail.com 10 Indonesia Mr. Cahyana Widyastama NGO / Field Indonesia widyastamacahyana@gmail.com 11 India Ms. Anne-Sophie Poisot FAO AnneSophie.Poisot@fao.org 12 India Mr. Konda Reddy Chavva FAO Konda.Chavva@fao.org 13 Italy Ms. Marjon Fredrix FAO Marjon.Fredrix@fao.org 14 Jordan Mr. Ashraf Alhawamdeh FAO Ashraf.AlHawamdeh@fao.org 15 Kenya Ms. Deborah Duveskong FAO Deborah.Duveskog@fao.org 16 Kenya Mr. Chrispin Mwatate Dalmas IIRR chrispin.mwatate@iirr.org 17 Lao PDR Ms. Vornthalom Chanthavong FAO/IPM/ PRR Programme Vornthalom.Chanthavong@fao. org 18 Malawi Mr. James Robert Okoth FAO James.Okoth@fao.org 36

42 19 Mongolia Mr. Kevin D. Gallagher NGO/Ex-FAO 20 Mozambique Mr. Eugenio Macamo Junior FAO 21 Myanmar Ms. Sandar Myo MFVP 22 Nepal Mr. Dilli Ram Sharma DOA 23 Pakistan Mr. Jam Muhammad Kalid NGO / SOFT khalidjam@hotmail.com 24 Peru Mr. Willy Pradel Caceres CIP w.pradel@cgiar.org 25 Philippines Ms. Almalinda Abubakar Ex-FAO alm_abubakar@yahoo.com 26 Philippines Mr. Audy Maagad Government audy.maagad@gmail.com 27 Rwanda Mr. Joseph Higiro RAB joseph.higiro@rab.gov.rw 28 Thailand Ms. Areepan Upanisakorn DOAE uareepan@yahoo.com 29 Thailand Mr. Jan Willem Ketelaar FAO/RAP Johannes.Ketelaar@fao.org 30 Thailand Mr. Marut Jatiket TEF marutj@thaied.org 31 Vietnam Mr. Ngo Tien Dung NGO / ICERD ipmppd@fpt.vn 37

43 Annex 2 Global Workshop on Impact Assessment and Monitoring and Evaluation of Farmer Field School programmes September 2018, Sukosol Hotel Bangkok WORKSHOP PROGRAMME DAY 0 16 September, Sunday Time Sessions/Objectives Content During the Arrival of participants Welcome note upon check-in day For those interested Gather in the lobby of Sukosol and meet participants that have arrived, then all free to go for dinner or rest DAY 1 17 September, Monday Time Sessions/Objectives Content Field visit Gather in the lobby of Sukosol, Departure from hotel 0745 Objective: Share and learn from Thai Travel from Sukosol hotel to Prathumthani province experience and lessons learned on Visit NFE Agroecology Rice FFS and Chumchon, Wat monitoring and evaluation of FFS programmes at the local (and national) Tamlae Thong School level Visit NFE FFS and Ruamjai Prasit School LUNCH BREAK and travel to Ayatthaya province Field visit Objective: Share and learn from Thai experience and lessons learned on monitoring and evaluation of FFS programmes at the local (and national) level Meet with Young Smart Farmers products, marketing and networking Visit Ancient City 1700 Departure for Bangkok Putting up posters in the meeting From 6 pm onward at the Kamolmart Room on 6 th floor. room 38

44 DAY 2 18 September, Tuesday Time Sessions/Objectives Content Welcome Remarks Background Workshop objectives and agenda Introduction to the organizing team Expectations for the workshop Getting to know each other quick round Feedback on Field Visit on Day 1 Observations and issues on the field visit, including those on M&E Main Impacts of FFS programmes in Different Parts of the World Sharing of impacts of FFS programmes and experiences on impact assessment Objective: Share and learn from country experiences and lessons learned on impact assessment of FFS TEA BREAK Main Impacts of FFS Training in Different Parts of the World...continuation As we go along Overview of Global Synthesis document, main impact domains, challenges for M&E Henk van den Berg Case study presentation: Asia s Impact Assessment of FFS programmes on Pesticide Risk Reduction Training - Gerd Walter Echols Questions and discussion LUNCH BREAK Main areas of impacts M&E visual (target, mandala figure 13) to guide discussions, and to be detailed during the workshop. Three phases: Monitoring and Evaluation of FFS programmes Objective: Share and learn from different projects and programmes and lessons learned on monitoring and evaluation of FFS programmes TEA BREAK Monitoring and Evaluation of FFS programmes..continuation Evaluation of the day Dinner cruise Depart hotel 1700 Areas/domains of impact, Indicators M&E tools used (qualitative, quantitative), tools to be added requirements/actions to systematize and mainstream this in FFS programmes) Field Programme experiences - Indicators What are main indicators that are used to measure impact What are indicators that should be added Challenges related to indicators What indicators are priorities, essential? What has been done to measure different impacts (link to poster presentation, etc) Feedback small group discussions, updating the M&E visual Dinner boat cruise from 19:30 21:30. The Grand Pearl boat 39

45 DAY 3 19 September, Wednesday Time Sessions/Objectives Content Recap of Day 2 Summary review of Day 2 & highlights Reflection Monitoring and Evaluation of Field Programme experiences - M&E tools for impacts FFS programmes..continuation What tools are used to measure different impacts, advantages and disadvantages? Are there additional tools to think of? What are essential, priority tools? Qualitative vs Quantitative? Challenges? Elaboration of tools for M&E TEA BREAK Feedback, adding to the visual M&E framework Panel discussion focused on: What is it that we want to change? What are issues that we often run into, the questions that are always asked? What each FFS programme should put in place for efficient M&E What information and tools are important LUNCH BREAK Monitoring and Evaluation of FFS programmes..continuation Field Programme experiences - M&E tools for project management and FFS quality, and Learning What data are collected on FFSs that are relevant for project planning and project management? How are data collected? Developments? Are there additional tools to think of? What processes are in place and what tools are used to ensure quality of FFS? What mechanisms to correct/improve? How to systematize lessons learned during implementation? feedback on M&E for programme management and quality FFS TEA BREAK Overview where are we, parking lot issues Evaluation, feedback on the day 40

46 DAY 4 20 September, Thursday Time Sessions/Objectives Content Recap day Monitoring and Evaluation of FFS programmes..continuation Field Programme experiences - requirements to mainstream Requirements and action points on how to strengthen M&E in FFS programmes feedback on requirements to mainstream, adding them to the visual M&E framework TEA BREAK Synthesis of discussions on M&E, action points for follow up after this workshop Finalizing the M&E framework and toolbox action points, timeline Overview of the global synthesis study on FFS Overview of the global synthesis study on FFS....continuation LUNCH BREAK :30 Wrap up and feedback Global Synthesis study main elements, current status Feedback on synthesis study Next steps to finalize the global synthesis Open discussions Follow-up actions on M&E and impact assessment, what can we do after the workshop Discussion on what is next for regional networks (?) Feedback, evaluation of the workshop 41

47 Annex 3 FIELD VISIT A field trip was organized on Monday, 17 September Its objective was to share and learn from the Thai Education Foundation experience and lesson-learned on monitoring and evaluation of FFS programmes at the local level. Thai Education Foundation The Thai Education Foundation (TEF) is a non-profit organization working to improve education in Thailand. It considers education first and foremost a continuous process guided primarily by the participant-learners themselves. To optimize this learner-centered approach in a rapidly changing national context, formal, non-formal and private-sector institutions must continually develop diverse and responsive capacities and programs. Through linking innovative field projects, training and policy development support, TEF works to make innovative, practical and effective contributions to this end. Its activities are in the following areas: Educational reform and decentralization Local capacity building Non-formal, adult and primary education, incl. primary school FFS Learner-centered, experiential education and action research Training and curriculum development and facilitation Environmental education and ecological agriculture Management information systems and educational technology Growing from numerous collaborative efforts between the Thai Ministry of Education, World Education, Inc. and FAO since 1990, TEF is now expanding the program throughout Southeast Asian countries through The Field Alliance. 1 st Stop: Chumchon Wat Tamlae Thong School, Klong 10, Prathumthani Province The school is an example of the Rural Ecology and Agricultural Livelihoods education programme of Thai Education. It has the objective (1) to create awareness of the importance of agrobiodiversity; (2) to promote conservation and sustainability; and (3) to reduce the use of toxic pesticides through the application of ecological agricultural practices. It is being applied with farmers through FFS as well as with students in formal and non-formal education systems. 42

48 This FFS was set up at the request of Prathumthani Province. The training of trainers took place over 4 months on 5 days/month. Meetings with farmers were organized at the community temple during religious days. The FFS designed field experiments in the school garden and provided training in agro-ecology and in raising crickets. Furthermore, the students conducted surveys on pesticide use, storage and disposal. They also participated in provincial exhibition. Last year s activities focused on community based chemical management to protect children from pesticide risks. With the help of test kits, they measured the pesticide contamination in vegetables in nearby markets and in school lunches. They found residues below MRL in 100% of the lunches and 98% of student s urine samples. Blood tests at the local health center showed that only 7% of the samples were negative, while 60% showed cholinesterase inhibition at critical levels. Furthermore, they found that the meat in the market was contaminated with formalin. As a result of these activities, representatives from four ministries will meet soon to discuss the situation. Comments: The programme created community awareness about pesticides and other household chemicals Farmers explained, that they are now collect and burn pesticide containers; wear more protective clothing, reduced their pesticides and use more biological products As a result of the FFS, farmers found that they can grow lotus flowers for temple offerings After the initial FFS focus on IPM, more attention is now given on health and environmental issues There is a serious concern about the future of farming because only few students become farmers and the land base is shrinking The school continues the activities within its own budget, but needs technical support 43

49 44

50 2 nd Stop: Ruamjai Prasit School, Klong 14, Prathumthani Province The small rural school with mostly Muslim students participates in the Royal Programme on School Lunches and has many agricultural activities. For about one year, they have conducted a non-formal education FFS with students and farmers for sharing knowledge and providing support to the school. They raise catfish in a pond and have set up comparison plots of transplanted and broadcast rice. Furthermore, they grow mushrooms, raise eels and produce earthworms in the school s green houses. Comments The school supports the King s strategy for national self-sufficiency Schools want to attract children to food production as a countermeasure against big industrial farms After graduation from school, most students work in factories (pay more) and only have time on weekends to attend their farms. Only about 20% become full-time farmers. Successful school programs cooperate with local agencies The Ministry of Agriculture provides marketing platforms to support young farmers 45

51 3 rd Stop: Young Smart Farmers Mon Nueng Village, Apoe Laad Buang Luang, Ayutthaya Province Under the Thai Government s Smart Farming Policy, the authorities aim to raise the per capita farmer income to more than $12,000 per year by Among the different official activities is a programme called Young Smart Farmers (YSF) which targets people aged 17 to 45 years old. It started in 2013 and aims to attract 2 million young farmers, but so far only about have been involved in 9 regions. There is the expectation that an entrepreneurial farmer or Smart Farmer will be especially appealing to the younger generation. It could attract new innovative minds to the agricultural sector where before it was seen as a dead-end career. Young people who may have been brought up in a city environment and became familiar with IT tools could easily transfer their skills to the farming environment if they preferred living in a more rural setting. The workshop participants visited a group of Young Smart Farmers at the farm of one of their members. They had also invited colleagues from neighboring provinces to display their products and join the discussions. The farm owner greeted the visitors and explained all the products she produced on her farm: rice for seeds, greenhouse vegetables (salad, Kai-lan, PakChoi) mushrooms on rice straw, sunflower oil, ducks, chicken eggs, and edible freshwater snails. Together with other farmers, she exports vegetables to Europe. She does not use pesticides, and her fertilizer comes from cow and buffalo manure, and from vermicomposting. 46

52 Like many of her colleagues, she learned her agricultural technologies from other farmers through the YSF network. In this government-sponsored programme, 25 farmers are selected per province and joined with 5 previous farmers who act as mentors. The group first receives and 4-day orientation training in management, technical training, entrepreneurship, and marketing. During the first year, they regular meet for exchange visits and study tours. There are two types of YSF farmers: those who come from their parent s farm, and young professionals who want to try their luck in agriculture. In the group visited, half the members had degrees and experience in various fields: electronics manufacturing, communications, interior design, MSc. in retail, engineer at an R&D section, logistics manager, teacher or a marketing expert for Coca Cola Company. The reasons for leaving their previous jobs were that they wanted to be nearer to their families, expand the family farm, escape a boring job or seeking more quality of life. Besides the minimal support they receive from the government, the YSF mostly rely on its own members. The internet is very important to them for on-line exchanges, finding answers, planning together and marketing their products. Some even use drones for monitoring crop development and irrigation. For starting their new agricultural enterprises, they found it easy to get bank loans and rent land. Through the extension system, they get assistance with quality certification, opportunities to show their products in big cities and contacts to companies which export to Europe. Their biggest challenges are natural disasters such as flooding or storms, diversifying their range of products, focusing on value-added products and overcoming family resistance to go organic. 47

53 4 th Stop: Wat Chedi Hoi Bo Ngoen Village, Lat Lum Kaeo District, Pathum Thani Province Before heading back to Bangkok, the group stopped at the Thai temple, whose stupa was built out of million-year old giant oyster shell fossils which were found in the surrounding area. 48

54 Annex 4 POSTERS Angola 49

55 Bangladesh 50

56 Brazil 51

57 Burkina Faso 52

58 Cambodia 53

59 54

60 East Africa 55

61 Ethiopia 56

62 India 57

63 Indonesia 58

64 Jordan 59

65 Kenya 60

66 Lao PDR 61

67 Malawi (1) 62

68 63

69 Malawi (2) 64

70 Mozambique 65

71 Myanmar 66

72 Nepal 67

73 Pakistan 68

74 Peru 69

75 Philippines 70

76 Rwanda 71

77 Thailand 72

78 Vietnam 73

79 Yemen 74

80 Annex 5 PRESENTATIONS Global FFS Review: Some thoughts on M&E by Henk van den Berg 75

81 76

82 77

83 78

84 Case Study of Asia s Impact Assessment of FFS Programmes on Pesticide Risk Reduction by Gerd Walter-Echols 79

85 80

86 81

87 82

88 83

89 84

90 85

91 86

92 87

93 88

94 89

95 90

96 91

97 92

98 93

99 94

100 Global FFS Review: Overview of Findings by Henk van den Berg 95

101 96

102 97

103 98

104 99

105 100

106 101

107 102

108 103

109 104

110 FFS Impact Meta Study Eastern Africa by Crispin Mwatate 105

111 106

112 107

113 108

114 109

115 110

116 111

117 112

118 113

119 Annex 6 GROUP SESSIONS 1 st Group Session: Updating the M&E Mandala Level: Skills Social Increased percentage of mixed male female FFS (in place where this was not acceptable) Level of group cohesion Decision making capacity at household level Number of farmer groups that continue to function after FFS Level of ownership of learning agenda Decision making capacity at group level Natural Knowledge of understanding agro-climate systems Access to agro-climate information AESA Knowledge to measure costs and benefits Financial Negotiatio n skills Knowledge of financial tools Human Technical skills Problem skills Level: Outputs Social Group cohesion Contacts and linkages with other farmer groups Financial Percentage of costs of production/reduction 114

120 Increase of learning platforms Decrease incidence of conflict in community (farmers/pastoralists/water users etc.) Increase in school attendance in FFS families Number of groups with minutes of records of meetings and actions implemented Natural Knowledge on practice and technology Informed use of inputs (externa) Ecosystem management Identification of friends and foes Human Observations Education Level: Outcomes Social Value chain participation and improvement Decrease in domestic violence in FFS families, less conflict Organization and networking Increase in active FFS participation Empowerment Changes in attitudes and perceptions Decision making Gender inclusion Trust Level of organization Natural Value chain participation and improvement Intensification and diversification Productivity Reduction in hazardous chemicals and inputs Environment and hazards Diversity of farming system Area under conservation Adoption of practices and technologies Financial Number and amount of credit loan savings Functional market linkage established Economic analysis Market access increased Market and marketing skills/added value Input cost reduction Increase accumulation of assets Increase in production quantity and quality Human Innovation Farmer to farmer learning Level: Impact Social Number of laws, community decision passed in consultation with farmer groups Percentage of male vs female in leadership in the community Quality of lives improves Institutionalization Local policy Leadership and community development Financial Percentage increase in income Source of income Income level Percentage profitability increase Increased income 115

121 Increase number of social events in community, sense of belonging mixing between social groups Non-formal to formal education connections Natural Adaptation of best management practices biodiversity Human Food security Quality of nutrition Balance in role sharing Access to information and communication technology 116

122 2 nd Group Session: Causal Chain Level: Skills Social Human Natural Financial Social Skill Output Outcome Impact Super Impact Group mobilization Analytical skills Questioning /awareness norms Decision making Presentation skills Technical skills Trust & accountability Responsibility Changes in behaviour, attitudes and perception Increase in space of women in household decision making Change in power and roles in households Empowerment Increase in leadership and Community Development Good cooperation in community and household Quality of lives, inclusive and equitable societies 117

123 Human Skill Output Outcome Impact Social skills: communication, networking, leadership skills Financial skills: cost management, financial skills No. of solved problems (pest, water, finance, etc.) Extent of problem analysis conducted by FFS Extent of technical skills acquired No. of decisions made by farmers Extent of observation skills acquired Analytical skills No of sharing/training by FFS farmers to non-ffs farmers No of farmer facilitators Farmer adopted good practices No of FFS farmers graduated Improved cropping pattern Access to diversified food Reduction of % pre and post harvest loss % increase in yield No. of certified farms (GAP, PGS, etc.) Improved food security at household level in specific community Natural Skill Output Outcome Impact AESA Knowledge, practice and technology of crop and animal rearing Knowledge of Environment & Hazards Knowledge of understanding agro-climatic systems Identification of friends and foe Access to agricultural information Adoption of best management practices Informed use of inputs Diversification and intensification of individual farm level Area under conservation Diversity of farming system in landscape Productivity Improved Ecosystem Services Financial Skill Output Outcome Impact Negotiation skills (human skills) Cost-benefit analysis Innovation; e.g.with premium quality Promotion with group Improved production in quality and quantity Market research about demand Response to market demand; work with market agents; buyers that want the quality you have Increased income Sustainable Market: portfolio of potential buyers 118

124 3 rd Group Session: Indicators and Methods of verification Social Human Natural Financial Domain: Social Description Indicator Method of data collection Skill Questioning/ awareness, norms Presentation skills Decision making Proportion of participants taking steps towards cultural norms No. of times having presented 119 Observation Focus Group Discussion FGD KII Records

125 Group mobilization Output - Trust Responsibility Accountability Outcome Change in behavior and attitude Increase in space of women in household decision making Change in power and role in household Impact Good cooperation in community and household Leadership and community development Inclusive and equitable societies Quality of lives through empowerment No. of participants contributing to key decisions made Group norms - roles and responsibilities - contributions - minute of meeting - No of conflicts - Accuracy of group records Proportion of participants taking steps toward changing behavior Level of satisfaction among women in household decision making Level of Satisfaction No of women in leadership position Level of satisfaction of community about their involvement in society development Observation Interview Group record Observation Group records Observations Interview with members Group records Observation FGD KII? Household interview Exercise (PRA) Interview Exercise (PRA) Interview with village leaders/local authority and women members Interview PRA Social mapping Domain: Human Domain Indicator Method Skill Observation skill Output FFS farmers/group graduated Outcome Farmer adopted No of AESA conducted No of analyses conducted by farmers No of observation of pest/ predators No of decisions made by farmers Decision pattern No of farmers graduated as per graduation certification criteria (..) No of action plans prepared by graduated farmers No of post FFS activities established No of farmers adopted/adapted practices in their farm 120 FFS Record book, observation, SS Interview, baseline FFS record book (ODK/e-FFS) Record on action plan open discussion, FGD, baseline FGD, IDI individual action, field visit, record, KI, baseline

126 /adapted good practices Impact Improved human capability Strengthened production capacity of farmer Domain: Natural Description Indicator Method of verification Skill Farmers develop capacities to observe, analyze and decision taking concerning agroecological management and knowledge Output Adopters of Good Agricultural Practices in the productive systems Outcome Balanced productive systems Integrated and diversified systems Impact Sustainability of farm and landscape No of practices disseminated and adapted by FFS farmers % of yield of crop of FFS farmer against with baseline = Extent of decision making by FFS farmers # of participants # of thematic areas # of curriculums # of practices Inputs Kg/ha # of crops # of production systems # of animals/plants Yield (kg/ha) # of farming systems Area under farming system (sq km/?? Quality of soil (% o.m.) Water quality Diversity/area Insects/animals/plant Farmer register records Survey Records Survey Survey Baseline survey Records Survey Lab analysis ~ Field observation Record, survey (Ass t), FGD, IDI, baseline Advantage Easy to collect Application in different moments Application in different moments Trust information ~ ~ Limitation Inconsistent participation in FFS High cost Quality of the information High cost Quality of the information Access to lab services Costly Complex information Domain: Financial Area Indicator Method of verification Improve production Reduced cost of production increased efficiency and yield Increase % premium grade products Increased profit Records Interviews Research Reporting (often measured in % increase) Market research Volume Demand identified # of potential buyers Promotion # of buyers contacted # of fairs or market places attended Negotiation skills # Contracts finalized 121

127 Cost-benefit analysis Outputs/Outcome Sustainable Market: Premium quality in response to market demand Response to Market demand Analyses completed Commodity identified ($) Percentage of premium product Certification of premium product Percent production increase of demanded product Production and timing 4 th Group Session: M&E tools for project management, FFS quality and learning Group 1 Group 2 Group 3 Group 4 Group 1: Social Group Who What How Farmer group - Agronomy AESA 122

128 Facilitators Technicians - Pest and diseases - natural enemies - climate conditions - Field condition Food security - diversity - Meals - Hunger gap Pre-test Meteorological data - rainfall - -humidity - -temperature Cropping data - types of crops - Social data - index of population - problems - qualification Equipment Economic - productivity - labor Infrastructure - roads - transport - irrigation (2) Additional tools - IT: phones, mobiles - plant clinics (3) Tools in place Interview Survey (4) Mechanisms Systematization Group discussion M&E Feedbacks Workshop for trainers to share experiences Follow up Meteorological station PRA Questionnaire Records ~ ~ Group 2 Data to be collected How Process Group/Small Organization/Village FFS record book Online: Open Data Kit, e-ffs Before, during FFS Facilitator/extension gender worker/farmer/village village profile leaders/members, etc. host team profile farmer profile attendance 123

129 leadership Amounts of contents (2) Learning AESA, Observations weekly/fortnightly/monthly FFS plot farmer, facilitator AESA Experiments Field days Economic analysis, survey, studies, data box, before, during, after curriculum/enterprise, technical topics covered participation discussion, FFS hand book (2) facilitators record during and after - profile (3) Consultation of community (4) KAP of farmers before FFS pre-test, PRA, case study before FFS Group 3 What data How Who Additional tools Process/tools for quality + mechanism systematization Baseline participants/farms/landscape situation Process quality Impacts - indicators Survey/questionnaire formats/template Focus Group discussion Farmer Field Book Farmers Facilitators Supervisors Third parties Financial data app/farm economic Real time data app Geotagging/geotarging information Incorporate M&E strategies/ tools in ToT Planning/evaluation meeting with multi stakeholders: farmers, facilitators, super Meetings to different level: local, national, regional, international Tools for documentation Enabling environmental Organizational information Feedback evaluation Group 4 (financial group) Registry of FFS: participants, FFS, field Focus group Stories of changes comments on changes Questionnaire Registry for evaluation: (by facilitators and farmers) Evaluation design at project planning during FFS implementation capacity of evaluators (weakness) summarize data 124

130 5 th Group Session: How to strengthen M&E? Group 1 Group 2 Group 3 Group 4 Group 1 Requirements for strong M&E system Clear framework with checklists Manual with clear protocols to follow: design, sampling, time, frequency Roles and responsibilities of different stakeholders Dedicated resources (human and financial); M&E team and space Capacity building for M&E and project teams Harmonized data collection forms Data management system Feedback loop to the communities; peer review Regular review and update M&E system Linkages to relevant institutions, national level data collection (SDGs, ministries) Mainstreaming 125

131 Factor potential indicators into project framework during program formulation Clarify relevance of each indicator in project framework Work plan and time frame, incl. end of project Build requisite capacity for importance of proper data collection Ensure feedback loop in place to different stakeholders Group 2 Strengthening FFS M&E Designing appropriate M&E framework Defining scope of M&E Develop customized M&E tools/methods Improve technical capacity of extension workers on M&E Close follow up of M&E framework Establish and conduct platform for M&E sharing and learning Organize and conduct peer monitoring Advocacy and fund mobilization Conduct impact evaluation at different levels Communicate major impacts of FFS evaluation Group 3 Requirements Dedicated M&E staff Simple framework? for M&E and more specific tools for? More specific tool for tech Simple generic tool for M&E Make better use of smartphone and applications for monitoring Coordination committee for M&E Budget for Framework Development workshop Budget for M&E Incentive and appreciation Institutionalization of M&E Regular feedback and learning sessions Group 4 Advocacy & fund mobilization Communicate major impacts of FFS evaluation Conduct impact different level Feedback on results of M&E 126

132 6 th Group Session: Action Plans Group 1: Action Plan Elaborate mandala Develop/revise participatory M &E framework (if there is a need) Share/fit participatory M&E tool kit Share findings/outcome of the workshop with M&E colleagues/partners Incorporate M&E module into ToF, master trainer curriculum Share presentations and reports to workshop participants Establish/strengthen e-network Organize follow-up workshop/learning events Group 2: Next steps Draft toolkit circulated for our input Exploring and FFS M&E app for basic info Collecting M&E data and materials for sharing Identify and apply for HQ funds for workshop M&E training for master trainers /Whatsapp to connect M&E staff; who will facilitate? Personal commitment for sharing M&E actions among group members 127

133 Group 3 Distribute synthesis of workshop and discuss with FFS community to get additional input Expedite sharing the toolbox to assist with national level briefings National workshops on the M&E (FAO funds) Use FFS platform to populate dashboards with FFS results from the field 128

134 Annex 7 EXPECTATIONS, FEEDBACK AND FINAL EVALUATION Expectations Group 1 Clarify the impact assessment: timing, etc. Harmonized framework for doing M&E in FFS Action plan to conduct FFS M&E Indicators in detail (gender sensitivity, Group 3 The Future of FFS & its sustainability Understand the importance of impact of the FFS Capitalize the main information Project-friendly Impact Assessment tools Group 4 To.. learn success and failure of different (1) tools of M&E; (2) approach of FFS; (3) implementation strategies..across countries share knowledge and experiences global network get enough inputs for developing FFS M&E toolkit 129

135 Recap of Day 2 Questions Key learning Impact chain analysis using mandala How to develop FFS indicators for impact Self-assessment tools are critical Identifying indicators and method of data collection for FFS Institutionalization of FFS is not an end in itself. FFS programs need to adapt to emerging needs and varying environments We have to think about IT tools in FFS M&E FFS programs need to improve developing M&E to measure impact FFS don t get markets for farmers but give farmers as a means to understand the dynamics Need for additional tools for improving follow-up of FFS e.g. IT Questions From the point of view of the expert, what should be the impact in the different domain? Impact goal of Human domain / Outcome objective of Human domain I still don t know how to do an impact assessment study! Methods? Where to start? How to Plan for Impact measurement when FFS is based on projects with short cycles. Are we going to solve the problem of funding to do the impact assessment after closing the projects? FAO has never give us money to do it once the project have come to an end. How do we simplify FFS monitoring? What are the basic datas to be collected apart from AESA datas? What about M&E at program level? We have only focused at group level How to identify the evaluation channels within the FFS life cycle Rapid assessment of impact of FFS. Need to be designed especially for short term projects 130