Thank you for inviting me. My role within the organisation has been to deal with the demands of the PPA. Core to this has been Value for Money (VfM).

Size: px

Start display at page:

Download "Thank you for inviting me. My role within the organisation has been to deal with the demands of the PPA. Core to this has been Value for Money (VfM)."

Austen Ray
5 years ago
Views:

Notes: Thank you for inviting me. My role within the organisation has been to deal with the demands of the PPA. Core to this has been Value for Money (VfM).

1 Notes: Thank you for inviting me. My role within the organisation has been to deal with the demands of the PPA. Core to this has been Value for Money (VfM). It is fair enough to say that VfM wouldn t be so high on our agenda if it was not central to our donors - DFID is leading the way on this. However, DFID is not imposing an approach it s pushing us to come up with one. WWF-UK defines value for money as the efficient and effective use of resources to support the best possible results for people and nature. We have a responsibility: To spend the money entrusted to us by our supporters and funders in the best possible way; That our actions are beneficial to the communities we engage with; And a desire and drive to make the biggest bang for every Pound we spend. Because of the new methodologies that VfM brings to conservation, there is plenty of room for innovation in making decisions but also in representing our results and convincing people that the environment is important. And it is this last aspect I will be looking at at. Innovation can mean different things to different folks but it has certainly being a new way of thinking for WWF-UK. 1

VfM explored through relationships between inputs (cost) and impact (difference made) The 4 E s framework of economy, efficiency, effectiveness and equity (see box) is commonly used by NGOs when

2 VfM explored through relationships between inputs (cost) and impact (difference made) The 4 E s framework of economy, efficiency, effectiveness and equity (see box) is commonly used by NGOs when analysing VfM. A number of useful resources are available to support practitioners in reviewing the economy, efficiency and effectiveness of their programmes (see WWF VfM Guidelines). However, much less has been written on integrating equity. 2

3 This is somewhat of an aside to what I am going to be presenting next but nevertheless very important. The first graph. 1. DFID s approach to Value for Money (2011) - A useful starting point that breaks value for money down by the key steps in the causal logic of the intervention process Focus on this sequence of logical steps have inadvertently led to disproportionate focus on inputs, economy and efficiency The second captured a lot of my thinking and was prepared by Coffey (a consultancy firm) which dealt with the M&E of the PPA. 2. Value for money evaluation framed by value first approach that tests the extent to which decisions about value and its generation drive the choice of inputs and costs 3

4 I am most going to be focusing on the latter but the former has helped us describe our approach and work on addressing some of the gaps A useful management tool was developed by the PPA Vfm Learning Group and helped WWF frame its approach and highlight it s area of weakness 4

5 The checklist was produced by the Value For Money PPA Learning Partnership, a group of 20 NGOs who mapped their own value for money activity and used the results to compile a list of what they saw as the key steps for an organisation to address in managing and measuring Value for Money. 5

6 Last year I commissioned a study with NEF Consultancy to assist programmes with undertaking a VfM analysis for themselves learning by doing. The first phase was to introduce the key methodologies and we principally covered these four here and then let the team pick one and run with the analysis. CEAI MCA - Permitted a comparison between previous and future strategies. Celtic Seas - BER (Basic Efficiency Ratio) - For limited resources, permitted a more holistic view of the programme than other methodologies. FOCAC - BER (Basic Effectiveness Ratio) Choice guided by conversations with NEF Consulting based on programme s needs. 6

7 The Celtic Seas Partnership (CSP) project was one of 5 WWF projects chosen to take part in a pilot Value for Money study. The aim of the study was to demonstrate that the CSP project (and project activities) has been Value for Money. The main research question for the study was: 'which engagement activities in the CSP project represent the best value for money (level of investment vs impact)?'. Data and information was collected through stakeholder surveys, and analysing the results of the surveys against the budget for each objective. 7

8 Basic Effectiveness Ratio (BER) was chosen as the best for CSP. BER is a simple form of value for money analysis that compares the outputs of different activities with the costs required to complete those activities. For this analysis, we have adapted the ratio to focus on outcomes and objectives (rather than outputs). In this BER analysis we consider how cost effective the achievement of different objectives are relative to the investments required to achieve each objectives i.e. complete the mixture of activities that support that objective. With an impact and investment score for each objective, these scores can be mapped onto a four quadrant graph (see Figure 2) to indicate the cost effectiveness of each objective relative to one another. To arrive at an impact score, the objectives were evidenced through use of indicator surveys from stakeholders who have been engaged with the Celtic Seas Partnership project. 8

As part of this analysis an impact map was developed for the project, and can be seen in Figure 1. The impact map shows the long-term objectives, with the medium-term outcomes listed below these.

9 As part of this analysis an impact map was developed for the project, and can be seen in Figure 1. The impact map shows the long-term objectives, with the medium-term outcomes listed below these. The long-term and medium-term outcomes and objectives feed into the immediate aim of the project (by the end of the project), which in turn, feeds into the overall aim (beyond the project life). For this analysis, we have adapted the ratio to focus on outcomes and objectives (rather than outputs). The yellow stars on Figure 1 indicate which objectives this BER will focus on. In this BER analysis we consider how cost effective the achievement of different objectives are relative to the investments required to achieve each objectives i.e. complete the mixture of activities that support that objective. Key steps include: - Putting together an impact map - Understand who the stakeholders are - Design a survey explore the before and after; and attribution - Try to map budget against activities and activities against outcomes - Quadrant graph (see below). 9

In addition to the indicator questions, asked in a before and after format, additional questions were posed, asking stakeholders the extent of any change that the Celtic Seas Partnership might have

10 In addition to the indicator questions, asked in a before and after format, additional questions were posed, asking stakeholders the extent of any change that the Celtic Seas Partnership might have been responsible for. We produced three separate online surveys, each tailored specifically to policy makers, task group members and general stakeholders. The questions asked in the online surveys can be seen in Appendix 1. The surveys were sent out to a wide variety of stakeholders from different sectors (listed in Appendix 3) and they were given 16 days to respond. To arrive at an investment score, WWF created a matrix whereby they placed activities and work streams against objectives sometimes placing the same activities across multiple objectives. They then placed budgets against each of the activities and sub-divided those budgets where an activity was deemed to contribute to more than one objective. From this point, budget estimates for each objective could be derived and a percentage of the overall budget for each objective calculated. Before related to before the Celtic Seas Partnership project started (January 2013). After referred to the completion of the most recent Celtic Seas Partnership process (up to June 2016). We tailored the surveys because not all the questions were relevant to all stakeholders. The before and after indicators allow a baseline to be collected. This was particularly useful because baselines were not collected on these indicators at the beginning of the Celtic Seas Partnership. The full project budget was used (including the budgets from partners) because the figures for spend to date were not available at this time. 10

11 11

12 - Investment is measured on the vertical axis and impact on the horizontal axis. This means that objectives located in the lower right hand quadrant of the Figure (high impact and low investment) could generally be deemed better value for money. Conversely, those objectives located in the higher left hand side of the figure (low impact, high investment) could generally be deemed less value for money. Those in the remaining two quadrants produce a mixed Value for Money story. 12

13 (Using an indicator baseline of 0 for some of the Objectives can give a misleading attribution figure. For example, the indicator for awareness to the project activities the project activities didn t exist before the project, so a score of 0 was given. Measuring this against the figure for their awareness to the activities now is always going to be higher, giving it a high attribution score.) What we learned from the process: It s useful and straight forward to carry out The graph allows you to make management decisions about investment vs impact It took much longer than expected to finalise the indicators and data collection tool for this study. Send out to a larger sample More useful when activities are complete 13

14 The Coastal East Africa programme is trying to promote sustainable forestry in the region. Multi-Criteria Appraisal (MCA) is relevant only if wanting to compare across interventions, most notably at the appraisal stage. It is, like SROI, stakeholder driven. It allows to rank interventions according to different criteria, but nonetheless does not require monetization of non-marketed goods. As such, it escapes some of the inherent biases of SROI and CBA. It can be particularly useful when choosing the type of approach a donor/ngo wants to choose after having determined the type/sector of an intervention, e.g. investigating the tradeoffs of a rights-based approach versus a classic programmatic approach. Finally, it is useful for analysing competing interests and objectives, specifically for what refers to access to scarce resources. MCA provides a process that leads to logical well-structured decision-making. The stages of an MCA are as follows: Stage 1: The decision context is clearly defined and understood, scope is established and those who will be affected by the decision (the stakeholders) are identified. Stage 2: The scenarios are defined what options could occur and the criteria by which we will determine the impacts of these scenarios are defined. Scenario 1: Sustainable Forest Management (SFM) but no focus on mechanisms to attract external financial investment locally controlled forest. Scenario 2: Scenario 1 but with a focus on promoting financial investment locally controlled forest. 14

15 Scenario 3: SFM plus a focus on promoting financial investment privately controlled forest. Stage 3: We collect evidence to quantify what changes under these scenarios. Stage 4: Understand the relative importance of these changes to the different stakeholders. Stage 5: Finally, we bring together all this information in a matrix that ranks the different scenarios for different stakeholders. Stage 6: This information is used to inform decisions. 14

16 Analysis can reveal trade-offs likely to provide overall consensus between groups 15

17 16

18 17

19 18

20 19

Introduction As mentioned, we are encouraging PPA programme to undertake deeper analysis for their programmes. WWF-Colombia have piloted a Cost-Benefit Analysis (CBA).

21 Introduction As mentioned, we are encouraging PPA programme to undertake deeper analysis for their programmes. WWF-Colombia have piloted a Cost-Benefit Analysis (CBA). The CBA was undertaken for the purpose of evaluating a Climate Change Adaptation and Biodiversity Conservation project, through livestock farming reconversion, in the Amazonian Andes (Region of Putumayo). This analysis takes a case study approach. Instead of executing a high level CBA for the programme as a whole, the analysis deals with a specific project. To give you a little background, the region of Putumayo is in the south of Colombia bordering Ecuador and Peru. There, livestock farming has been identified as the main driver of deterioration of the Amazon ecosystem, putting at risk not only the environment per se but equally economic activities relying on these critical ecosystem services. The over-arching objectives of WWF s intervention have been twofold: Prevent the further degradation of ecosystems and subsequently improve them Increase the revenue of farmers through 1) intensification of production via the application of new production techniques and management; and 2) the maintenance of those ecosystem services that are critical to the sustainability of production practices (e.g. preventing further soil degradation and water contamination). 20

22 Cost-benefit analysis is a methodology which aims to compare the cost of an intervention to its impacts or benefits. When impacts are not economic, then non-market valuation techniques need to be used in order to express these non-monetary impacts into monetary terms. Rather than empirically deriving the value of ecosystem services, this CBA uses existing figures and transfers them to the site in question. This approach is by and large referred to as benefits transfers. A lot of those we used came from The Economics of Ecosystems and Biodiversity (TEEB) research. This table summarises the key benefits considered in the analysis. The cost figures, which are not presented here, considered both WWF budget and the investment of households. 21

23 The cost and benefits occurring into the future (assumed 10 years) were discounted to represent their present value ie the worth in today s money. These results suggest that: Benefits significantly outweigh costs, even when accounting for beneficiaries inputs into the projects. If trusting these findings, the returns on investment are high: for each 1 invested in the intervention, between 10.5 and 14.6 are generated. The fact that benefits outweigh costs by such a margin means that even if the benefits are overestimated and/or costs under-estimated, positive returns should still be expected. 22

24 Equally important is to grasp where the value is generated. In this case, the bulk of benefits are created in the form of improved climate and hydrological regulation services. Critically, the project equally improves the economic prospects of beneficiaries (17%) But we need to remain critical as to whether these figures are reliable. 23

25 This briefing note presented the results of a first attempt to apply a CBA methodology for programmes and projects implemented and financed by WWF-Colombia. Overall, this research is very encouraging. WWF-Colombia managed to apply a CBA methodology in a relatively comprehensive way. It also considered the ES literature and managed to insert environmental impacts into a CBA balance sheet. The results of the CBA are equally encouraging for the Climate Change Adaptation and Biodiversity Conservation Project in the Amazonian Andes. Despite caveats, the returns that the project delivers are positive and high. This intervention appears highly effective in delivering the intended changes. The critical review of the CBA process and findings pointed to points of improvement for future CBAs. The two major points of improvement required are a) a better consideration of the magnitude of change (in this study the assumption was that ; and b) more consideration of the counterfactual (even in a qualitative form) e.g would part of the impact would have happened anyway? 24

TOOL #57. ANALYTICAL METHODS TO COMPARE OPTIONS OR ASSESS

TOOL #57. ANALYTICAL METHODS TO COMPARE OPTIONS OR ASSESS PERFORMANCE 1. INTRODUCTION A crucial part of any retrospective evaluation is the assessment of the performance of the existing policy intervention.