Module j:\cc69\nmm\website docs\new\pm&e core course\module 3 logic models 2\module 3 - logic models ii nov 2004.doc

Transcription

1 Module Module 3 Project Identification and Design: Developing Logic Model - Part 2

2 4-2

3 3-3 TABLE OF CONTENTS 1. CONVERTING AGREED INTERVENTIONS INTO A RESULTS CHAIN CONVERTING THE RESULTS CHAIN INTO PLANNING LANGUAGE... 7 TIPS ON WRITING RESULTS STATEMENTS LOGICAL FRAMEWORK ANALYSIS ASSESSING RISKS AND ASSUMPTIONS Killer Assumptions and Pre-conditions DEVELOPING INDICATORS Indicators for Results Direct Indicators Proxy Indicators Quantitative Indicators...18 Qualitative Indicators Combination of qualitative and quantitative indicators Targets and Milestones DEVELOPING INDICATORS FOR RESULTS Develop Evaluation Questions Brainstorm several indicators Set targets Assess the Quality of the Indicator and Select Indicators Document the Rationale Means of Verification EXERCISE 3.1 CONVERTING A RESULTS CHAIN TO AN IUCN PLANNING CHAIN EXERCISE 3.2 ANALYSING CONDITIONS NECESSARY FOR SUCCESS EXERCISE GENERATING INDICATORS OPTIONAL EXERCISE 3.4 COMPLETING A LOGIC MODEL... 28

4 3-4 Notes:

5 3-5 Learning Objectives: At the end of Module 3, participants will be: Able to convert a results chain into a planning chain Familiar with logical framework analysis and their relationship to logic models Familiar with risks and assumptions Able to generate and assess critical risks and assumptions Familiar with the concepts of indicators, targets, milestones, benchmarks, performance criteria and means of verification. Able to relate indicator types to result levels. Able to develop types of indicators (quantifiable, qualitative, etc.). Able to develop targets, milestones Approximate Duration: 6 hours Overview: In this module: Presentation: Converting interventions into a results chain and a planning chain Exercise 3.1: converting a results chain to a planning chain Presentation: Logframes Presentation: Analysis of Risks and Assumptions Exercise 3.2: Analysis of Risks and Assumptions Presentation: Generating Indicators Exercise 3.3: Generating Indicators Optional Exercise 3.4: Completing Logic Models

6 3-6 Notes:

7 3-7 Basic Concepts Logic Models 1. Converting Agreed Interventions into a Results Chain In the previous module, one or more project interventions were developed. The next step is to construct a results chain for these interventions, or to refine the results chain already developed during. If the problem analysis approach was used we need to arrange the interventions into a results chain, following the definitions provided in exhibit 1.2. The simplest way to create a results chain is to convert the means-ends diagram into a results chain using the following method. Starting from the top of the means-end model: The core problem or highest level of suggested solution becomes an impact statement. This will also correspond to the changes expected to occur in the condition of people and the environment. The next order of suggested results becomes a statement of suggested outcomes, corresponding to the changes we foresee in the institutions and their systems to promote improved management and use of natural resources. The next level becomes a description of immediate effects. These changes we expect to see immediately in people's attitudes, levels of capacity and knowledge as a result of the project. Next level (causes by now) will correspond to the outputs that must be delivered. The lowest level corresponds to the activities that must be implemented to get the outputs outlined above. 2. Converting the Results Chain into Planning Language Often project plans are written in a language slightly different form the results chain. Once we have a refined results chain, we may need to convert this to a planning language that is understood by the donor(s) who are likely to fund the project. As explained in Module 1, different donors often use different terms for the same concept. Exhibits 3.1 and 3.2 give examples of the languages used by some of the donors who fund IUCN projects. You will notice that most of the terms have four levels of results, while the results chain has five levels. This is because outputs, the deliverables occurring when we implement activities, are not usually included in the planning chain. For example, if a project is involved in training staff, the number of people trained (output) is an indicator of the training having happened. It will therefore not appear on the planning chain. Improving skills, instead of people trained, is the correct statement in planning language. Similarly, if a project is involved in formulating a management plan, the management plan document (output) will be the indicator of the formulation having happened and will not appear on the planning chain. Once we remove the outputs level from the results chain, the levels equate better with the planning chain, and transformation becomes easier (Exhibit 3.3 below).

8 3-8 Exhibit 3.1: Planning terms used by different donor agencies 1 IUCN NETHERLANDS EUROPEAN UNION WORLD BANK Long Term Objective Development Objective Overall Objective Development Objective Project Objective Immediate Objective Project Purpose Project Devel. Objective Specific Objectives Outputs Results Outputs Activities Activities Activities Activities Exhibit 3.2: More Examples of Planning terms used by different donor agencies 2 CIDA USAID NORAD IUCN Long Term Result Strategic Objective Goal Goal Medium Term Result Intermediate Result Purpose Purpose Short Term Result Project level Interm.Result Outputs Results Activities Activities Activities Activities 1 Source: Woodhill, IUCN 2000 Core PM&E Concept 2 Source: Woodhill, IUCN 2000 Core PM&E Concepts

9 3-9 You will notice that some donor agencies such as Netherlands, World Bank and NORAD use the term output in the third level of the planning chain. The meaning of the word output in such cases is the same as intermediate effects, the short-term changes occurring as a result of the project, rather than the usual meaning - deliverables occurring directly from implementation of activities. For example, improvement in skills rather than number of people trained in the previous example. IUCN advocates the use of Goal Purpose Results Activities chain whenever possible. Taking the IUCN preferred planning chain; the impact becomes the goal statement. The outcomes and the intermediate effects are analysed together to form two levels a project purpose statement and the results. The activities correspond to activities. Exhibit 3.3: Results versus Planning Language Planning Language Results Goal Impacts Purpose Outcomes Results Immediate Effects Activities Activities Inputs Inputs 3-56 Tips on Writing Results Statements Goal: This is the desired long-term change expected in the situation the project is influencing. It corresponds to impact, and it is the longer-term, higher order changes that should eventually be the results of the project. The project is expected to contribute but does not take FULL responsibility for achieving the goal. Other organisations contribute to it as well. The goal statement should be written in the present or past tense, as though it has already happened.

10 3-10 Purpose: This is the actual expected contribution of the project towards the ideal situation described in the goal. It corresponds to a synthesis of outcomes and immediate effects. It is the highest level of result that should occur as a direct consequence of interventions during the life of the project. The project therefore commits to achieving this contribution within an agreed timeframe and budget. The purpose should be presented in a clear succinct statement that clearly synthesizes all the results/objectives of the project, and has the potential to inspire the partners. In addition, it should be formulated and stated in a manner that allows identification and measurement of outcomes. It should also clearly identify the target groups and/or geographic area to be covered by the project. Finally, it should be written in the present tense or present continuous tense. Results These represent specific objectives or statements of management strategies to implement the purpose. They correspond to intermediate effects. Results always present changes in a real situation (ask yourself, who and what will change?) Actions to be done by the Project (i.e. train, research, promote ) should not be written as results: these are activities. Result should be written in past or present tense. In addition to the results identified during the planning, remember to include a result on project management, such as The project effectively and efficiently managed. This result will allow you to plan and budget administration tasks, co-ordination of meetings, supervising visits, monitoring and evaluation activities, etc. As far as possible results should be: SMART. This means they are Specific, Measurable, Achievable, Relevant (to the project purpose and goal) and can be achieved with a specific Timeframe. Activities are the do-ables, the points of expenditure. 3. Logical Framework Analysis A Logical Framework Analysis, often abbreviated as LFA, is an analytical planning tool that can provide: A logic model that includes performance measures A tool to organise thinking A flexible results-driven planning tool Methods that are widely accepted Methods that can accommodate new tools Projects are implemented in dynamic situations and adaptive management require that plans be constantly updated. LFAs provide a tool to organise and present the project plans in a visual, manner that is easy to update. Because they allow presentation of the various project components in summary form, LFAs make it possible to display the relationships between the various parts, making analysis easier.

11 3-11 Exhibit 3.4: Components of LFA Intervention logic Indicators Means of verification What the project will Evidence of Where the information do to bring about achievement that can to verify achievement change. An outline of be objectively verified will be found the theory of action Risks and assumptions Conditions necessary for success - Factors outside the control of management that can influence achievement of results Unfortunately, like planning language, LFA language can be confusing. This is largely because different donors use different terminology for the same concepts. (Exhibits 3.1 and 3.2). Since IUCN works with a range of donors, it is important that project managers understand the concept in order not to be confused by the label. 3.1 Assessing Risks and Assumptions Risks and assumptions are the set of conditions that must be present in order for the successful implementation of the project and the achievement of its results. They are the set of conditions that must be true in order for the results chain to work (exhibit 3.5 for examples). For example the conditions necessary for: Awareness activities to be implemented and products delivered Awareness to lead to change in attitudes Change in attitudes to lead to improved performance Improved performance to lead to improved management Improved management to lead to improved conservation status of a forest resource. Risks and assumptions are usually outside the direct control of the project management. Exhibit 3.5: Examples of Assumptions along the Results Chain A project dealing with restoration of a forest ecosystem might have the following as part of its results chain. Activity Output Intermediate effect Training community members on improved tree planting and husbandry X number of people trained X number of hectares replanted Community with skills of tree husbandry Risks and Assumptions That people attend training That weather patterns are favorable and support regeneration Targeting: That the people who attend the training are the same ones actually involved in tree husbandry

12 3-12 Outcome Impact Community forests being managed in accordance with the forestry husbandry guidelines Increased tree density, increased number of species - in other words a healthy forest That land tenure system will support tree husbandry practices That communities accept principles advanced in the training That the right species have been selected and increase in density, cover and index is equal to improvement in health status. Risks and assumptions relate to the hypothesis on which the logic is build. In other words, the reason we think that doing one thing will lead to achievement of another. Some of these conditions relate more to the implementation process, such as people attending training, while others relate more to the theory of action, such as That land tenure system will support tree husbandry practices or That communities accept principles advance in the training. Other examples of risks and assumptions include: Some of the implementing partners loose the will and capacity to implement the project. There is a drought and the agricultural production project has no irrigation. Political instability forces general disturbances and donor withdrawal of funds. Political support for the program changes due to such things as an election of a new government, change in government priorities. It is important to stress that the risks and assumptions are out of control of the project. They may be external or internal factors related to the theory of action, but they are usually out of the control of the project. The analysis of risks and assumptions completes the analysis of the project theory and provides important information on which to base decisions about which results to include, modify or discard. Two simple methods can be used to decide which conditions have serious implications for the selected result or intervention: the simple matrix and the sophisticated algorithm (Exhibits 3.6 and 3.7).

13 3-13 Exhibit 3.6: Simple Analysis of risks and assumptions Likelihood of the condition holding true Severe Minor Category B Category A Category C Category D Consequences for meeting objectives Low High Both methods assess the likelihood of the condition to hold true and the severity of the potential consequences to the success of the project if the condition failed to hold true. The simple matrix loosely classifies the condition into categories A D. Category A: Conditions with a low likelihood of happening and of minor consequence if they failed. Category B: Conditions with a low likelihood of holding true, but severe consequence if it failed Category C: Conditions with a high likelihood of holding true and severe consequence if it failed (preconditions?) Category D: Conditions with a high likelihood of holding true and a minor consequence if they failed to hold true. The importance of any one condition varies from one project to another, depending on the project context and realities. For example, the issue of weather or rainfall in the example above can be an important factor for one project and unimportant for another project. If the project described above was dependent on natural rainfall rather than irrigation, then weather patterns become a serious issue. If the project is in the semi-arid areas such as agro-ecological climatic zone 5 where rainfall reliability is rather low, the likelihood of this factor being realized is high, and the implications on the survival of seedlings is severe. So for this project, the factor would be in category C. If on the other hand the project was in a high rainfall area such as in agro-ecological climatic zone 2 where weather patterns are more reliable, the likelihood of this factor being realized is lower than the case described above. The implication for the survival of the seedlings is likely to be severe, though probably not as severe as in the case above. In this case the factor would be placed in category B.

14 3-14 Exhibit 3.7:The Sophisticated - Assumption Algorithm Guiding questions Probable responses Probable Action Question 1: Is the external factor important? No Yes Discard issue Proceed to question 2 Question 2: Will it be realized? We are almost sure it will happen We think it might happen, but we are not sure Include as an assumption and implement project or part thereof Include as a risk. Implement project if we are risk takers, monitor to adjust management We are almost sure it is unlikely to happen The project is not technically feasible proceed to question 3 Question 3: Is it possible to redesign the project to influence the external condition? Yes No Redesign the project and add results and activities to influence the factor Revise the project to reflect what can be achieved or discard entirely

15 3-15 The assumption algorithm is a sophisticated tool for determining whether to include an assumption in the PPM or not. Its use revolves around three questions: Is the factor important? Is likely to be realized do we know the probability of it being realized? Can the project be redesigned to take care of the factor if it is so important that ignoring it would seriously compromise the success? For example a project that aims to improve the living standard of people might have the following as part of its results chain. Level in the result What Risks and assumptions chain Impacts Improved livelihoods That the rate of inflation is lower than the rate of increase in household income Outcomes Increased household income Immediate effects Better performing natural resources based businesses Output X number of people with improved skills for managing natural resource based enterprises Activity Train entrepreneurs on sustainable methods of harvesting alternative income generating products Brainstorm a list of conditions and pick one for further analysis example, the rate of increase in income is higher than the rate of inflation Question 1: Is the condition important: Ask yourself whether the condition is necessary for the successful implementation or for the achievement of results. If it is not important, it is discarded and the analysis moves to the next condition. If it is important, we move to question two. Our condition is important, so we proceed to the next question. Question 2: Is it likely to be realized? Ask yourself if the condition is likely to become a reality. In our example, we are asking whether the rate of income can indeed rise faster than inflation or in other words, whether the rate of inflation is likely to be lower than the rate of increase in income. There are 3 possible responses to this question: We are almost sure it will. We are not sure which way it will go, but it is likely to be faster. We are almost sure it will not. The response to this question will, of course, depend on the context of the project.

16 3-16 Some scenarios to consider: A project in a growing economy: If a project is being implemented in a country with a growing economy, we are almost sure that the rate of increase in income will be higher than the rate o inflation. In this case, our theory of action is based on the belief that we can raise income faster than inflation. We therefore keep the condition as an assumption and monitor its effect on the success of the project. If the project is highly successful, it can be used to explain the success. A project in a stable economy: If the project is being implemented in a stable economy, it would be likely that the rate of inflation is lower than the rate at which the project causes incomes to increase. However, we cannot be sure that this will happen. This means that our theory of action is based on a condition whose probability of occurrence we are uncertain. If we like taking risks, we will go ahead with the project. So we include the condition so we can monitor its behavior and adjust project strategy accordingly. If we do not like to take risks, we probably should either redesign the project or add parts that reduce the risk, or not implement it at all. In this case we proceed to question 3. A project in a declining or unstable economy: If on the other hand the project is being implemented in an economy with a clear pattern of decline where the rate of inflation is very likely to rise faster than the rate of increase in income, then we are almost certain that the project cannot cause incomes to rise faster than inflation. This means that some of the impacts will not be realised, even if the project successfully implements all its activities, produces all its outputs, realises all the immediate effects and outcomes. In its current form, this project is not viable and so we proceed to the last question. Question 3: Is it possible to re-design the project in order to influence the external condition: The purpose of planning is to achieve all the project results. If it becomes clear that a condition outside the control of the project managers will interfere with the achievement of any level of results, then the obvious thing to do is to redesign the project to include activities that influence it. If this is not possible, then the project is not technically feasible. This is often referred to as a killer assumption. Killer Assumptions and Pre-conditions A killer assumption is a high threat assumption that will cause a project or its parts to fail if it is not addressed. When a killer assumption is identified, the project needs to be redesigned to include activities or results to influence it. If this is not possible, the project has either to be discarded because it is not technically feasible, or revised to reflect what can be achieved with the present limitations. Killer assumptions should be project preconditions. For example, That communities are willing to participate in a community-based natural resources management project. Classification of risks and assumptions is not a scientific exercise. The point is to be able to use the matrix to determine whether to include the condition in the project-planning matrix (PPM) in order to monitor its effect on the performance of the project, and adjust management strategies accordingly. The importance of this analysis is to clarify the necessary conditions for the project to achieve its desired results. This implies monitoring. Ideally the risks and assumptions included in the PPM should be monitored for their effects on the project performance. This is particularly important for learning projects. Used well, it improves the chances of designing realistic projects.

17 Developing indicators Once the intervention logic has been identified and conditions necessary for success analysed, it is time to generate indicators. Although indicators have been defined in many ways, most definitions concur that an indicator provides evidence of change, or signs that the conditions the project is trying to improve have changed. They also provide evidence of the progress of project implementation and attainment of results. Indicators help to answer the question, How shall we know that outputs have been produced and that results have been achieved. The specific question depends on the level of the indicator in the LFA (Exhibit 3.8). Exhibit 3.8 Indicators along the LFA Level of result in the LFA Goal Purpose Result/Objective Activities Indicators Evidence of long-term changes in the condition (impacts). Note that projects contribute to impacts, so indicators would show contribution rather than wholesale change. Evidence that outcomes have been realised (institutions and systems are changing) Evidence that intermediate effects are happening (people are changing) Evidence that activities were implemented and outputs were produced - deliverables. Activity indicators are the outputs, the deliverables that should be evident when an activity has been implemented. For example, the number of people trained or of workshops held, study tours conducted, number of seedlings planted etc. These are simple and need no further discussion. Indicators for Results Unlike indicators for the activity level, results indicators (intermediate effects, outcomes and impacts) are more complex. They can be direct or indirect (proxy), qualitative, or quantitative. Direct Indicators Direct indicators provide the most direct evidence or measure. For example, if a project result deals with the transfer of sustainable farming methods, it might be stated as, Farming practices improved. The most direct indicator for this result would be the percentage of farmers who have successfully adopted x number of specific sustainable practices. For this indicator to make sense, two concepts need to be defined: sustainable farming practices and how many specific sustainable practices constitute sustainable farming practices. Proxy Indicators It will not always be possible to measure change directly, either because information is not available or because the cost of collecting it is prohibitive. In these cases, proxy indicators have to be used. For a project dealing with the restoration of a watershed, for example, reduced erosion might be chosen as an indicator. Because it is difficult to measure the level of erosion, the level of sedimentation in the river may be used as a proxy. Where it is difficult to collect information about the evolution of a community's wellbeing, the number of improved houses is

18 3-18 often used as a proxy. This requires defining improved houses which can range from tin roofs to bricks, depending on the community. Infant mortality has also been used as a proxy for changes in health levels. Proxy indicators are very often used in conservation projects where short-term interventions are adopted to change long-term processes. In a project seeking to improve livelihoods and the quality of a local forest, for example, it may be very difficult to identify meaningful impact indicators that can be monitored during the lifetime of the project. In such cases, proxies such as the percentage increase in household income and number of seedlings surviving beyond a certain age may be used as proxies as they demonstrate the contribution rather than the attainment of the ultimate impact, which is in line with the project's contribution to the goal. Quantitative Indicators Quantitative indicators refer to evidence that can be demonstrated in countable measures. The number of seedlings surviving beyond a certain age or the percentage of people with zinc roofs, bicycles, or a combination of both. Other examples are hectarage of forest under joint forest management, the percentage change in species richness index, or the percentage change in tree density, etc. Qualitative Indicators A qualitative indicator is a non-quantitative evidence of change. For example, if our result is improved capacity of local organisations to manage their resources, it might be very difficult to find a number that shows this. In such a case, we might choose a qualitative indicator such as change in the strength of local organisations and then define ten characteristics, for example, that explain what we understand by organisational strength. We may then establish a scale like the one below 3 : Very weak - those exhibiting none of the 10 characteristics Weak those exhibiting less than 3 of the characteristics Relatively strong those showing 4-6 of the characteristics Strong those showing 7-9 of the characteristics Very strong those showing all the 10 characteristics This process is referred to as quantifying qualitative indicators. Here, we used a scale (10 characteristics) and a range (very weak to very strong) to quantify the indicator. Another example is the rate at which improved farming practices are adopted, as an indicator for improvement in farming practices. But adoption might mean very different things to different farmers, so that a simple number may not correlate well with actual improvement in farming practices in general. A non-quantifiable measure such as the extent to which farmers pick and practice the better practices may be more informative because it would tell us whether farmers are adopting or adapting the improved practices. Once again, we have to define characteristics of a successful adoption, using the number of practices adopted and the level of practice. 3 Source Alexandro Imbach 2001, Finding the Way

19 3-19 Combination of qualitative and quantitative indicators A combination of qualitative and qualitative indicators is often used. For example, the percentage (quantitative) of organisations in the strong to very strong category (qualitative) or the percentage of staff with positive perceptions on improvement of performance as a result of training. These classifications are not mutually exclusive since one indicator can fall under several categories, be direct and qualitative, a direct proxy, etc. The important thing is that the evidence can be independently verified in a non-subjective manner. Targets and Milestones These two concepts are closely related to indicators and we need to understand them. A target is the level of achievement that a project expects to reach within a specific time frame. A milestone, on the other hand, is a critical point in the life of a project by which time key activities should have been completed and/or key targets reached. Setting and monitoring milestones and targets helps to keep a project focused on achieving its results and provides an overview of progress. Indicators sometimes include targets and/or milestones that may become part of the objective/result statement or put in a separate column. Different donors often have different requirements. The important thing is to understand what targets and milestones are and how to treat them. IUCN recommends including the target as an integral part of an indicator because the process of determining targets acts as a reality check on the planning. While results/objectives are usually written in a narrative, generic manner, the indicators that incorporate targets specify the dimensions to make the result clearer. For example, a result/objective might be Strengthened capacity of community-based organisations to manage natural resources. The related indicator helps set the dimensions by explicitly expressing, Four local organisations in three villages have the capacity to engage in improved natural resources management practices." Analysing quantitative and qualitative aspects of the indicators provides an excellent opportunity for in-depth discussion about what the project really means. When addressing complex issues - usually summarised in such vague words as equity, sustainability, or participation - it is important that everyone be clear on what others understand when they hear and act on these concepts. Indicators are therefore useful for defining with precision the scope of the project results/objectives, and for assessing the project's performance.

20 Developing Indicators for Results Several steps: Develop evaluation questions Brainstorm several indicators Set targets Assess the quality of the indicator and select indicators Document the rationale Develop Evaluation Questions A practical suggestion for developing meaningful indicators is to begin by developing evaluation questions. Take each result and ask, How will we know (or demonstrate to others) that the changes stated in the results were achieved satisfactorily? Who will change as a result of the project's activities? What will change because of the project? This will help to limit the selection of indicators to those that will demonstrate change that can be expected from the relevant result. Brainstorm several indicators After having identified the evaluation questions, brainstorm to determine several indicators that can provide answers. Set targets Targets are set to specify how much change can be realistically expected as a result of a project intervention. We need to work with others to set realistic targets. The first step is to project the future trend, then add a value that we think can be achieved and establish the final performance target (see below). We then need to define the progress starting from the baseline, which is the value of an indicator at project year zero. This is often a problem because projects often have no resources to establish baselines. Example: A project has the result, Greater awareness of the role and importance of collaborative forestry management among all stakeholders. It will go through the following steps to set targets. Projecting the future trend Awareness would probably increase somewhat even without the project intervention, but by low value, say 10 %. Determining a value we think the project can attain say 60% Add it to the value that might be attained without the project intervention (10+60) = 70%; i.e. to say that awareness will increase by 70% in a certain period of time Define the progress from the baseline. This would be to say that awareness would improve from the current x% to (x+60)% in the specified time. However, we rarely know the value of x, the awareness level at project year zero, or have the money to establish them.

21 3-21 Another example: Goal: Improving conservation status of forest X. Indicator: Percentage change in species richness index (this might be one of many) Projecting the future trend Species richness index would probably increase somewhat even without the project intervention, but by a low value such as 5%. Determining a value we think the project can attain say 30% Add it to the value that might be attained without the project intervention (5+30) = 35%; i.e. to say that species richness index will increase by 35% in a certain period of time Define the progress from baseline. This would be to say species richness index would improve from the current x% to (x+35)% in the specified time. In some cases, the species richness index may be set to decline rather than increase without the project intervention. Here, the project would attempt to stabilise the rate of decline or to reverse it. The project staff must decide whether the index is likely to increase or decline on the basis of the situation analysis. Setting targets for such a case is outlined below. Project the future trend species richness index will probably decline by, say, 50% without the project intervention. Determine a value by which the project can realistically reduce the rate of decline, say 80%. Subtract from the rate at which it might be lost without the project intervention: (50-40) =10 (note 80% of 50 is 40). Define the progress from the baseline. This would be to say that the rate of decline of species richness index would be reduced from 50% to 10% in the specified time. Assess the Quality of the Indicator and Select Indicators Collecting information on indicators costs money. Each result indicator selected will need a monitoring action plan (see module 4). It is therefore very important to select a few robust indicators that will be monitored rather than a long list of indicators that will not. Selection should be based on the following criteria; Measurable Can it be measured, in quantitative or qualitative terms? Relevant Are the indicators relevant to the project purpose and goal? Attainable Are the results which the indicator seeks to chart progress realistic? Specificity refers to the ability of an indicator to measure progress towards the desired result. Available Is the data available at reasonable cost and effort?

22 3-22 Document the Rationale Once the indicators are selected, it is important to document the rationale used to select them as a future reference for project staff and others. In many projects, the first attempt leads to imperfect indicators that express the initial level of precision reached by the design team about the changes to be pursued. As the project evolves, so does the understanding of the reality in which the project works. As the logframes need to be reviewed periodically, various opportunities will arise to improve indicators based on the practical experience of the project. This is the essence of adaptive management. All project team members should be aware that the indicators are for the team, not for the donor or others. They are the team s way of establishing their own criteria for success (If we achieve such and such things, then we did it well). Therefore thinking, planning and deciding upon indicators is one of the best ways to use time in the design stage of the project and at the periodic reviews. Means of Verification Means of verification refers to the source of information on the indicators. It answers the question, where will the information on indicators be found? This is another way of ensuring that the proposed indicators are measurable and that the measurement is feasible. The means of verifying the indicators can range from using documents and reports to doing field verification and interviews with final users. In some cases, it may be necessary to make specific studies such as forest inventories or aerial photography studies to collect the required information. It is very helpful later to correctly identify these means of verification when the project monitoring and evaluation plan is being prepared and the project budget and staff requirements are being defined.

23 3-23 Exercise 3.1 Converting a results chain to an IUCN planning chain Time: 50 minutes Instructions: Working with the cause-effect model from Exercise 2.5: 1. Check to see if it is a refined results chain refine if needed (5-10 minutes) 2. Convert the refined results chain to the IUCN planning chain in accordance with instructions in this module (20 minutes) 3. Use the Worksheet 3.1 below. 4. We will discuss this in the large group (20 minutes). Worksheet 3.1 Level Statement Goal Purpose Specific Objectives (Results 1 n) Activities

24 3-24 Exercise 3.2 Analysing Conditions Necessary for Success Time: 60 minutes Instructions: 1. In groups, using the planning chain constructed in exercise 3.1, complete worksheet 3.2 on the next page (20 minutes) 2. Use the algorithm described in this module to determine which assumptions to include in the project and identify killer assumptions. (20 minutes) 3. We will discuss this in the large group (20 minutes).

25 3-25 Worksheet 3.2: 4. Identify risks and assumptions for all levels of the means-end model. Level of result Risks Assumptions Outcomes to Impacts What external factors might interfere with realisation of impacts What internal conditions must be present in order for outcomes to lead to impacts Intermediate Effects to Outcomes External factors that might interfere with realisation of outcomes What internal conditions must be present in order for outputs to lead to outcomes Outputs to Intermediate Effects What external factors might interfere with production of outputs What internal conditions are necessary in order for activities to produce outputs Activities to Outputs What external factors might interfere with implementation of activities

26 3-26

27 3-27 Exercise Generating Indicators Time: 1 hr 20 minutes Instructions: Working on the logic model from exercise 3.2, develop indicators using the following steps (40 minutes): Develop evaluation questions Brainstorm indicators Set targets and timelines Select final indicators Use the worksheet 3.3 below. We will discuss this in the large group (20 minutes). Worksheet 3.3 Evaluation question Indicator Goal Change Target Timeline Purpose Choose one result (specific objective) In addition identify the indicators for activities, remembering that outputs are activity indicators. Record them in worksheet 3.3b below (20 minutes). Activity Activity 1 Activity 2 Activity 2 Output

28 3-28 Optional Exercise 3.4 Completing a Logic Model Time: 45 minutes Instructions: Working with a partner, study the Results Chain and identify a) the conditions that are necessary for success between intermediate effects and outcomes, and outcomes and impacts, and b) outputs and activities for each intermediate effect. Use the worksheets below (20 minutes). We will discuss the results of the exercise in a large group (25 minutes). Worksheet: Logic Model Immediate effects Stakeholders with skills to enable them effectively engage in JFM Outcomes Impacts JFM plans formulated and implemented Knowledge base to support planning implementation of JFM developed Stakeholders with capacity to undertake improved forest management and utilisation Improved livelihood and status of forests in Brazil Policies that support JFM formulated 1-24 Stakeholders with awareness of the values and importance of JFM Global M &E Initiative A) Worksheet for Conditions

29 3-29 B): Worksheet for identifying Outputs and Activities for Intermediate Effects. Immediate Effects Outputs Activities Stakeholders with skills to enable them to effectively engage in Joint Forest Management Joint Forest Management plans formulated and implemented Knowledge base to support planning/ implementation of Joint Forest Management developed Policies that support Joint Forest Management formulated Stakeholders with awareness of the values and importance of Joint Forest Management.

30 3-30 Test your understanding 1. Differentiate between results chain and planning language 2. How do you convert agreed interventions into a results chain and later into a planning chain? Why is it important to do this? 3. What are the defining characteristics of the following: a. Good goal statement b. Good purpose statement c. Good results (specific objectives) d. Outputs e. Activities 4. What are the four columns of a logframe? 5. Differentiate between risks and assumptions. Why is it important to assess risks and assumptions during the planning process? What is a killer assumption? 6. Explain the use of the assumptions algorithm to a colleague 7. List the steps you would use to develop an indicator for a result 8. Use examples to differentiate between a. Direct and proxy indicators b. Quantitative and qualitative indicators c. Targets and milestones d. Indicators and milestones e. Indicators and targets 9. Explain the process of quantifying qualitative indicators 10. What are the important sections of a project proposal? Give a brief description of the contents of each section.