Definitions and coding book

Transcription

1 Kornelia Rassmann LinkedIn Tonya Schuetz LinkedIn CCAFS outcomes evaluation report Volume 4 Definitions and coding book Kornelia Rassmann, Tonya Schuetz, Philip Thornton, Laura Cramer May 2017 Preamble This is Volume 4 complementing four further volumes of an evaluation report commissioned by the CGIAR Research Program on Climate Change Agriculture Food Security (CCAFS). The evaluation was undertaken by the independent evaluators Kornelia Rassmann and Tonya Schuetz and supported by the CCAFS internal evaluation team led by Philip Thornton and Laura Cramer. It mainly used Outcome Harvesting (OH) but also elements from Impact Pathway thinking and Contribution Analysis to describe and analyze development outcomes that were directly or indirectly influenced by one of three CCAFS climate products the Climate-Portal, MarkSimGCM, and the Climate-Analogues tool. Volume 4 presents the terminology and the coding book used in this study. Definitions and the classification of outcome leads, researched cases and SMART outcomes were developed together with the CCAFS evaluation team and served four purposes: 1. Stimulate discussions within the evaluation team to clarify terms and get to a common understanding and terminology. 1. Help organize the data and allow efficient and quick access to specific categories of cases, to support discussion and interpretation of specific (groups of) cases and outcomes; 2. Serve as a framework for the interpretation of patterns observed in the survey responses, researched cases, and SMART outcomes (e.g., geographic spread of cases, types of outcomes); 3. Finally, the efforts put into defining a classification scheme were also considered useful for potentially informing future elements of CCAFS monitoring, evaluation and learning.

2 Use of the report The evaluation report comprises five volumes for different anticipated audiences Volumes 1 to 4 are available from the CGIAR website ( VOLUME / AUDIENCE Volume 1 CCAFS governance and management, funders, partners, stakeholders Volume 2 CCAFS governance and management, funders Volume 3 CCAFS governance and management, funders, partners, stakeholders Volume 4 CCAFS management team Volume 5 Internal (not published) CONTENT Main evaluation results Executive summary; background to the evaluation, its design and methodology (Sections 1, 2, 3); an overview on the cases that were researched in more depth (Section 4); answers to the evaluation questions based on the outcomes data (Section 5); insights from the evaluation process (Section 6); and recommended discussion points and opportunities (Section 7). Survey results and user perspectives Findings and conclusions from an online survey to the users of CCAFS climate data/tools, i.e. the potential contributors to development outcomes, conducted during the outcome harvest. Outcome stories The narratives developed together with our informants during this evaluation, describing who has been influenced to change in what way, and what contributed to these changes. Terms and coding book Definitions and classifications developed jointly by the internal and external evaluation teams to get a common understanding of terms; effectively organize and interpret the data; and potentially inform future CCAFS monitoring and evaluation. Databases, presentations, compiled secondary sources Excel databases developed during this study; PowerPoint presentations to guide discussions with the CCAFS evaluation team; interview data; researched literature. Kornelia Rassmann Tonya Schuetz Philip Thornton Laura Cramer page ii

3 Acronyms CC CCAFS CGIAR CIAT CRP CSA CSO EC GCM GIS INGO IFAD IP MEL NARES NGO R4D Sub-IDO ToC Climate Change CGIAR Research Program on Climate Change, Agriculture and Food Security formerly the Consultative Group for International Agricultural Research International Center for Tropical Agriculture CGIAR Research Center Climate smart agriculture Civil Society Organization European Commission General Circulation Model Geographic Information Systems International non-governmental organization International Fund for Agricultural Development Impact Pathway Monitoring, evaluation and learning National Agricultural Research and Extension Services Non-governmental organization Research for Development Sub-Intermediate Development Outcome Theory of Change Kornelia Rassmann Tonya Schuetz Philip Thornton Laura Cramer page iii

4 Table of Contents 1. PROGRAM THEORY 1 2. OUTCOMES 1 3. CLASSIFICATION OF TYPES OF DEVELOPMENT OUTCOMES 2 4. CCAFS LINK TO PROJECTS WHERE CLIMATE PRODUCTS WERE USED 2 5. PURPOSE OF RESEARCH USING CCAFS CLIMATE PRODUCTS 3 6. KEY GAME CHANGERS AND CONTRIBUTORS 4 7. TYPE OF CONTRIBUTION 5 8. GEOGRAPHIC PARAMETERS 6 9. CGIAR SUB-INTERMEDIARY DEVELOPMENT OUTCOMES (SUB-IDOS) REFERENCES 8 Kornelia Rassmann Tonya Schuetz Philip Thornton Laura Cramer page iv

5 1. Program theory Theory of Change (ToC) Impact Pathway (IP) Outcome Map Presents a hypothetical identification of the ways by which change is expected to occur from output to outcome and impact along an impact pathway. The ToC questions the assumptions about causality underlying the relationships between outputs, outcomes and impact. In TOC the assumptions present the mechanisms of change. (From CGIAR MARLO glossary) The causal pathway for a research project or program that outlines the expected sequence to achieve desired objectives beginning with inputs, moving through activities and outputs, and culminating in outcomes and impacts. Assumptions underpinning the causal chain and feed back loops are usually included. (From CGIAR MARLO glossary) Here: umbrella outcomes (see below) arranged following roughly the causal pathway of results, which may not necessarily have emerged in a linear way. 2. Outcomes Outcome area Outcome leads Upstream outcomes An area or theme where thematically related outcomes are harvested (e.g. within a specific program or project). Brief statements describing CCAFS results that potentially can be turned into SMART outcomes, but where specific, verifiable, plausible information is still missing. Outcomes that are more activity/output-near, i.e. occur earlier in the impact pathway and/or are more directly influenced by CCAFS research products. Downstream outcomes Outcomes that are more impact-near, i.e. occur later in the impact pathway and/or are indirectly influenced by CCAFS research products (e.g. through other outcomes in the causal chain). SMART outcomes Umbrella outcomes Observable changes in the behavior, relationships, activities and actions of individuals, groups, organizations or institutions that were influenced in a small or large way, directly or indirectly, intentionally or not by actors producing or using research outputs based on CCAFS climate data/tools. To qualify as an outcome the descriptions need to be SMART: Specific (formulated in sufficient detail), Measurable (providing objective, verifiable quantitative and qualitative information), Achieved (establishing a plausible relationship between the outcome and contribution), Relevant (presenting a significant step towards the impact that is strived for), Timely (emerging within the time period being evaluated). A generalized outcome summarizing several SMART outcomes that were clustered by similar type of societal actor and change. Kornelia Rassmann Tonya Schuetz Philip Thornton Laura Cramer page 1

6 3. Classification of types of development outcomes Development outcomes Stakeholders doing something differently with respect to CC-related knowledge, attitude, capacity, relationships, policy or practice, being (partly) influenced by evidence generated with the help of CCAFS climate data/tools; e.g. observed and/or documented changes in skills or in programming, investment, management, policy or agricultural practice relating to CC adaptation or mitigation; rather than the uptake of CCAFS climate data/tools by researchers to produce evidence or improve the tools and methods needed. Immediate level Behavioral changes in stakeholders concerning their financial or in-kind support of research output delivery, their level of awareness or institutional/personal capacity, or their training or advocacy strategy with respect to CC and CCAFS climate products. Resource-research Training-strategy Advocacy-strategy Awareness Capacity Intermediate level Resource- strat / impl Policy -agri /-env Ultimate level Practice -agri / -env Stakeholders such as donors or collaborating partners contributing financial or non-financial means to climate related projects using CCAFS data or tools. Stakeholders developing or adapting training/education agendas or strategies. Stakeholders adapting or developing new advocacy / lobbying strategies to better communicate climate adaptation and mitigation strategies. Stakeholders demonstrating changes in their awareness or attitude towards climate adaptation and mitigation strategies Increased institutional or personal capacity (knowledge / skills). Stakeholders using information influenced by CCAFS data/tools to take investment decisions, design, and/or implement new policies, plans and strategies furthering CCAFS goals with respect to CC adaptation and mitigation. Resource allocation: Stakeholders derive investment decisions for strategy development or implementation supporting CCAFS goals (e.g. development of national CC adaptation strategies, implementation of CSA programs, etc.) Policy formation: Stakeholders engage and develop decisions and/or new or different strategies (such as national action plans; research programs, corporate strategies; etc...) either for CSA programs (-agri) or for CC mitigation (e.g. GHG, LED) (-env) Behavioral changes in the lives of beneficiaries contributing to the three SLOs (reducing poverty; improving food and nutrition security; improving natural resources and ecosystem services) that were influenced at some stage and to some extent by activities, outputs and/or outcomes involving CCAFS data/tools. Practice changes: Beneficiaries implement or adapt strategies, i.e. do something different in agriculture (-agri) or environmental management (-env) with respect to CC adaptation or mitigation 4. CCAFS link to projects where climate products were used Link to CCAFS project In how far is the project where the CCAFS data/tool were used funded and/or known by CCAFS? 0 Project not funded and not known to CCAFS evaluation team 1 Project not funded but known to CCAFS evaluation team 2 Project funded and known to CCAFS evaluation team Kornelia Rassmann Tonya Schuetz Philip Thornton Laura Cramer page 2

7 5. Purpose of research using CCAFS climate products Purpose of tool usage Species/habitat distribution modeling CC impact study Climate risk screening Climate information services Identification of climate analogue sites What type of CC related project was the CCASFS product used for? Climate-Portal or MarkSimGCM employed to explore how climate change will influence the species distribution or habitat suitability for organisms including plants, forest species, reptiles, fish etc. for decision making in species, biodiversity or habitat conservation. Climate-Portal or MarkSimGCM used in studies assessing the influence of future climates on agricultural productivity, forestry, water availability, and other environmental or cultural ecosystem services, etc. Climate-Portal or MarkSimGCM used to identify risks for communities or development projects, e.g. to allocate additional budget for risk mitigation and disaster risk reduction. Use of the Climate-Portal or MarkSimGCM to retrieve high resolution data for national climate change reports to guide the implementation of agricultural plans, or downscaled seasonal forecasts to enhance early warning system, etc. Climate-Analogues tool used to project the future (or current) climate and rainfall predictions for a particular site and map where else in the world one could find a comparable current (or future) climate. These analyses serve different purposes, e.g. to identify - sites for farm-to-farm exchange visits - suitable agricultural strategies and CC adaptation planning - climate ready seed varieties - sites for crop evaluations trials Kornelia Rassmann Tonya Schuetz Philip Thornton Laura Cramer page 3

8 6. Key game changers and contributors Key game changers (societal actors) Contributor (change agents) The target audiences, i.e. the individuals, groups, organizations, institutions, who did something differently (partly) influenced by the activities, research outputs or outcomes of interventions to which CCAFS climate data/tools contributed to. Individuals, groups of individuals or organizations who influenced in a small or large way the behavioral changes of the key game changers. The following classifications were used for both key game changers and contributors Relationship to CCAFS CGIAR Partner Others Type of institution Academia Community CSO Government-nat / -loc / - reg / -glob Gov-inst Investor Private sector As defined in CCAFS online partner database: mes_tid=all&field_regions_tid=all CGIAR Centers and other CRPs Organization that either i) gets or gives funds to/from CGIAR, or ii) contributes inkind work in collaborations with CGIAR No formal relationship with CGIAR Universities Farmers, communities, smallholders. local, national or international active civil society organizations, i.e. institutions and initiatives or groups/networks of organizations that are not-for-profit and non-governmental (CSOs, NGOs, INGOs, including CGIAR Centers). Policy makers at local, national, regional or global levels (national Ministries, EC, UN, etc.. Government institutions or initiatives: National agricultural research extension systems (NARES), other national research institutions, national meteorological services, government initiatives and programs, etc.. Multilateral agencies and/or donors (World Bank, IFAD, Green Climate Fund, etc.) Companies, consultancies. Direct / indirect use of CCAFS climate data/tools and information/results generated with these Next-user / Development partner Primary next-user Secondary next-user Actors such as national research institutions, extension organizations, NGOs and others, which access CCAFS products directly or use research outputs generated with these. Next-users can be CGIAR or non-cgiar partners; research or nonresearch partners; they create an environment that enables the target impact for end-users; decision makers that we want to influence to achieve outcomes (adapted from Westermann et al., 2015). Researchers using CCAFS climate data/tools directly in their work to generate evidence or adapt the tools to better fit project needs. Development partners using evidence generated by others with the help of CCAFS' climate products, creating an environment enabling CC adaptation or mitigation. End-user The beneficiary population, usually quite massive, making it unfeasible for a project or program to work with them directly (Westermann et al., 2015). Kornelia Rassmann Tonya Schuetz Philip Thornton Laura Cramer page 4

9 7. Type of contribution Type of tool Portal Which of the three CCAFS climate products was used by the contributor? Use of CCAFS Climate Portal (2008, Citation of the statistical downscaled delta method dataset: Ramirez, J.; Jarvis, A High Resolution Statistically Downscaled Future Climate Surfaces. International Center for Tropical Agriculture (CIAT); CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS). Cali, Colombia. Citation of the spatial disaggregation method dataset Ramirez, J.; Jarvis, A Disaggregation of Global Circulation Model Outputs. International Center for Tropical Agriculture (CIAT); CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS). Cali, Colombia. MarkSim Use of CCAFS MarkSim (2009), new version MarkSimGCM (2013) ( Citation of the MarkSim pattern scaled dataset Jones, P.G.; Thornton, P.K.; Heinke, J Generating Characteristic daily weather data using downscaled climate model data from the IPCC's Fourth Assessment Report. CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS); Waen Associates; International Livestock Research Institute (ILRI); Potsdam Institute for Climate Impact Research (PIK). Jones, P. G. and Thornton, P. K. (2013) Generating downscaled weather data from a suite of climate models for agricultural modelling applications. Agric. Systems. 114 (2013) 1-5 Analogues Use of CCAFS Analogues tool (2011, Ramírez-Villegas J, Lau C, Köhler A-K, Signer J, Jarvis A, Arnell N, Osborne T, Hooker J Climate analogues: finding tomorrow s agriculture today. Working Paper no. 12. Cali, Colombia: CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS). Strategies used by contributors (CCAFS or non-ccafs) to influence key game changers Research-tool Research-evidence Capacity-ind / -inst Climate service Fundraising Advocacy Engagement Adapting CCAFS' data/tools to suit project needs, developing methodologies based on CCAFS' data/tools; testing results obtained through different approaches; pilot studies; proof of concept, etc. Generating evidence or recommendations based on research results (such as future climates for a particular area) using one or more of the three CCAFS tools for decision making or further use in advocacy, capacity building, engagement activities Capacity building mediated through one or more of the evaluated CCAFS tools or through sharing / communicating research evidence generated with CCAFS' data/tools, e.g. training of professionals in the use of the tools (workshops); holding or participating in meetings sharing research evidence; institutional capacity building through the establishment of partnerships (e.g. joint projects, collaboration) - leading to increased awareness and understanding of CC issues and integration of climate information into planning and decision making and/or to the use of CCAFS' climate data/tools; funding of further climate studies. Communicating timely and tailored climate information in an accessible format to communities and other stakeholders to allow e.g. suitable farming practices or risk management measures to be taken. Writing proposals, engaging with donors, etc. Developing promotional material (online or print non-scientific publications with repackaged, accessible climate information), running campaigns, media, etc. Engagement, lobbying, convening of stakeholders, bringing people together (similar or different stakeholders), farmer exchange visits, meetings. Strategies used by CCAFS to facilitate use of the climate products Data&tools provision Technical support Training Provision of accessible climate data or models through one or more of the evaluated CCAFS tools. Provision of manuals via website Technical support via s, Skype, etc. Workshops, collaborations Kornelia Rassmann Tonya Schuetz Philip Thornton Laura Cramer page 5

10 8. Geographic parameters Geographic location of tools users, potential reach of project outcomes Geographic location America-North-country America-Latin/Caribbean- Europe- Africa-North- Africa-West- Africa-South- Africa-East- Africa-Middle- Asia-West- Asia-Central- Asia-East- Asia-South- Asia-SouthEast- Oceania- Geographic reach Global Continent Region Country Sub-Country CCAFS core sites East Africa (EA) Ethiopia Kenya Tanzania Uganda West Arica Burkina Faso Ghana Mali Niger Senegal Latin America Colombia Guatemala Honduras Nicaragua South East Asia Cambodia Lao PDR Philippines Vietnam South Asia Bangladesh India Nepal Region plus name of country where research is being conducted (outcomes), or where informant is located (survey); Region classification follows Based on CGIAR s categories Additional category for projects targeting countries on several continents Corresponds to CGIAR s category Corresponds to CGIAR s category Corresponds to CGIAR s category Lump category for CGIAR s Province, District, Ward, Permanent agricultural trial site, River Basin, Village, Household Sites of CCAFS Climate-Smart Villages ( Kornelia Rassmann Tonya Schuetz Philip Thornton Laura Cramer page 6

11 9. CGIAR sub-intermediary Development Outcomes (sub-idos) Taken from the CGIAR Strategy and Results Framework For its Phase II, CCAFS has selected a number of sub-idos to which its work primarily contributes (bolded). These are the ones where CCAFS results and outcomes contributions will be monitored more rigorously than contributions to any of the other sub-idos. Small letters (a, b, c, d) indicate duplications of sub-idos. # Code Sub-IDO Increased household coping capacity to cope with shocks Reduced production risk Improved access to financial and other services Reduce Market Barriers Diversified enterprise opportunities Increased livelihood opportunities Increased value capture by producers More efficient use of inputs Reduced pre- and post- harvest losses, including those caused by climate change Closed yield gaps through improved agronomic and animal husbandry practices Enhanced genetic gain Increased conservation and use of genetic resources Increased access to productive assets, including natural resources Increased availability of diverse nutrient-rich foods Increased access to diverse nutrient-rich foods Optimized consumption of diverse nutrient rich foods Reduced biological and chemical hazards in the food system Appropriate regulatory environment for food safety Improved water quality 20 a Reduced livestock and fish disease risks associated with intensification and climate change 21 b Increased safe use of inputs Land, water and forest degradation (including deforestation) minimized and reversed 23 a Enhanced conservation of habitats and resources. 24 b Increased genetic diversity of agricultural and associated landscapes More productive and equitable management of natural resources Agricultural systems diversified and intensified in ways that protect soils and water Enrichment of plant and animal biodiversity for multiple goods and services Increased resilience of agro-ecosystems and communities, especially those including smallholders 29 c Enhanced adaptive capacity to climate risks 30 d Reduced net GHG emissions from agriculture, forests and others forms of land use 31 d A.1.1 (Reduced net GHG emissions from agriculture, forests and others forms of land use) 32 A.1.2 Increased above and below ground biomass for carbon sequestration 33 A.1.3 Improved forecasting of impacts of climate change and targeted technology development 34 c A.1.4 (Enhanced capacity to deal with climate extremes) 35 A.1.5 Enabled environment for climate resilience 36 B.1.1 Gender-equitable control of productive assets and resources 37 B.1.2 Technologies that reduce women's labor and energy expenditure developed and disseminated 38 B.1.3 Improved capacity of women and young people to participate in decision-making 39 C.1.1 Increased capacity of beneficiaries to adopt research outputs 40 C.1.2 Increased capacity of partner organizations, as evidenced by rates of investment in agricultural research 41 C.1.3 Conductive agricultural policy environment 42 C.1.4 Conducive environment for managing shocks and vulnerability, as evidenced in rapid response mechanisms. 43 D.1.1 Enhanced institutional capacity of partner research organizations 44 D.1.2 Enhanced individual capacity in partner research organizations through training and exchange 45 D.1.3 Increased capacity for innovation in partner research organizations 46 D.1.4 Increased capacity for innovation in partner development organizations and in poor & vulnerable communities Kornelia Rassmann Tonya Schuetz Philip Thornton Laura Cramer page 7

12 10. References CGIAR Strategy and Results Framework Gibbons S Understanding behavioural change: A guide for analysing factors influencing progress towards outcomes. Copenhagen, Denmark: CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS) Westermann O, Thornton P.K., Förch W Reaching more farmers: Innovative approaches to scaling up climate-smart agriculture. Working Paper No CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS). Kornelia Rassmann Tonya Schuetz Philip Thornton Laura Cramer page 8