CONTENTS Acronyms... i Introduction... 1 Understanding Value for Money Analysis... 2 Why use this Value for Money (VFM) guide?... 2 Defining VFM... 2 Who uses VFM analysis?... 3 What is VFM analysis?... 4 VFM process, factors and measurements... 4 VFM process descriptions... 5 How VFM analysis is used... 6 What is needed for a VFM analysis?... 7 Challenges to VFM analysis... 7 Conducting VfM analysis... 8 Approaches and tools for programme managers... 8 Using cost, performance and beneficiary data to determine unit-costs... 14 Using VFM measures to understand efficient and economic options... 17 Procurement analysis... 17 Elements of a procurement review... 19 Developing a VfM Framework for WASH programming... 21 Next Steps... 24 Frequently Asked Questions about VFM... 24
Acronyms ANC ASWA CO CPR DFID EOP FOB HH HR IP IPME ITN LLIN LOP MICS VDC VFM Ante-natal Care Accelerating Sanitation and Water for All Country Officer Cost-Performance Ratio Department for International Development End-of-Programme Free On Board Household Human Resources Implementing Partner Independent Process Monitoring Evaluation Insecticide-Treated Net Long-Lasting Insecticidal Net Life of Project Multiple Indicator Cluster Survey Village Development committee Value for Money
Introduction This Value for Money (VFM) process guide is produced by WYG International, which provides Independent Process Monitoring and Evaluation (IPME) for the DFID-funded Accelerating Sanitation and Water for All (ASWA) programme implemented by UNICEF in nine countries. The IPME assignment includes technical support related to monitoring and this guide is an outgrowth of VFM support to the UNICEF Nepal Country Office. The guide is intended to provide practical step-by-step instructions to development practitioners who wish to undertake basic VFM measurements. It combines brief educational notes about VFM measurements and its uses, with instructions and practical examples to illustrate the framework for data capture, VFM analysis, and high-level programme and implementing partner performance management. This guide draws upon, and is indebted to, recent Value for Money insights by the Mott Foundation, Oxford Policy Management, DFID, and the New Economic Forum. Value for Money (VfM) Process Notes P a g e 1
UNDERSTANDING VALUE FOR MONEY ANALYSIS Why use this Value for Money (VFM) guide? If you can t measure it, you can t manage it. 1 Value for Money (VFM) analysis in international development is an analytical tool to manage the performance of development interventions. It is not the only tool for performance management, but unlike other tools it links costs and budgets to expected and actual performance. By linking costs and performance, VFM analysis is able to delve into questions of economy, efficiency, effectiveness, and equity of benefit distribution. VFM differs from a cost-benefit analysis through its review of the processes that translate inputs into outputs and outcomes. This process-review aspect of VFM analysis compares elements of programme performance and regional differences that contribute to greater or lesser economy and efficiency of the processes leading to outputs. Consequently, while a cost-benefit analysis generates a quantitative ratio of cost to benefit over time; VFM analysis generates both a quantitative value and a qualitative value to determine the economy, efficiency, effectiveness and equity of an intervention by examining the processes that lead to output and outcomes. Both VFM analysis and a cost-benefit analysis are valuable tools to understand the value of development interventions and a cost-benefit analysis is sometime appropriate as part of a VFM analysis. These VFM Process Notes are intended to be a user-friendly guide to enable development practitioners to take the first steps toward high-level VFM monitoring and analysis in WASH programming. These VFM Process Notes provide a step-by-step explanation and guidance to programme management who want to use VFM approaches to manage programme performance. As such, this guide is a contribution to greater VFM knowledge and practice. This guide does not provide all the steps or measures for VFM analysis. Rather, it describes the steps and measures that were used in the Nepal ONE WASH Case Study. Illustrative examples use both WASH-specific data and data from other sectors. The reader will come away with a greater understanding how VFM can assist in managing and improving quality of programme performance. With that awareness and with the step-bystep framework provided in this guide, a programme or project will be able to effectively target both internal and external VFM expertise to measure and assess those factors that are essential to a programme s performance management. No programme needs to measure everything. Effective VFM measurement targets those factors that have the most impact on a specific programme s performance. The ultimate goal of this guide is to enable each country WASH programme participating in ASWA to establish a framework of VFM measures to manage their overall performance and the performance of their implementing partners. Defining VFM The British government s Department for International Development (DFID) promulgated this definition of VFM: Value for Money (VFM) is about maximising the impact of each (GB) Pound spent to improve poor people s lives. 2 1 Dr Peter Drucker, The Drucker Management Institute. 2 DFID s Approach to Value for Money (VfM), July 2011 Value for Money (VfM) Process Notes P a g e 2
The Australia Council for International Development (ACFID) defines VFM as: The optimum combination of whole-life cost and quality (or fitness for purpose) to meet the user s requirement. It can be assessed using the criteria of economy, efficiency and effectiveness, with some in the sector calling for the inclusion of a fourth E equity. 3 Similarly, the UK National Audit Office defines VFM as: The optimum combination of whole-life cost and quality (or fitness for purpose) to meet the user s requirement. 4 VFM, therefore, is quite different than simply achieving the lowest-cost provision of goods and services; VFM must ensure that the goods and services are acquired at the best possible quality and cost, and the use of goods and services must also meet the outcomes and expectations of the organization that is expending the resources. Some may ask if the above distinction between low-cost and quality is merely rhetorical, and if a donor will not almost always opt for the lowest cost option. While this is certainly a risk, it is our experience that donors primarily seek strong evidence often external evidence to support the selection of a chosen approach and that if evidence supports the choice of a higher-cost option, donors will support that choice. Maximising impact is not the same as minimising cost. In an environment of constrained funds for development, it is imperative for development practitioners to demonstrate that there is evidence to support the selected intervention. Providing such evidence mitigates against the risk of anyone choosing a less effective but lower cost option. The key is: provide the evidence to support your intervention. VFM is an approach that (1) seeks to understand the multiple factors that contribute to programme cost and performance; (2) measures the outputs, outcomes, and impact generated by inputs; and (3) analyses the spread of benefits across disaggregated beneficiary groups. Who uses VFM analysis? In daily life, we all make use of VFM analysis, though we seldom use that phrase. When you make a purchase, especially a larger purchase, you likely ask some very pertinent questions about VFM: Is the item to be purchased the appropriate quality, balanced with cost, to enable maximum impact? Is it priced competitively? Can I purchase the same item for less cost from another supplier? (Economy); Once I make the purchase, am I reasonably sure the product will work well? Will it continue without major breakdowns and unexpected costs? (Efficiency); If the product does what I expect, will it do so repeatedly? Will the product deliver the results I expect? (Effectiveness); Is the product worth the purchase price? Will the results I get from the product improve my life? (Effectiveness); Who will benefit from the purchase of this product? Are the benefits spread across all the intended beneficiaries? Are there other people in our household with other, more pressing, financial needs? (Equity). In the private sector, business owners make decisions about where and how much to invest by asking questions about how to obtain the greatest return on investment for the business. 3 Australian Council for International Development (ACFID), ACFID and Value for Money Discussion Paper, September, 2012. 4 UK National Audit Office, 2001, Getting value for money from procurement: How auditors can help, National Audit Office & Office of Government Commerce, London Value for Money (VfM) Process Notes P a g e 3
In the development sector, we do the same. Our bottom line is not profit as in a business. Our bottom line is results that improve lives. Value for Money in the development sector is getting the greatest improvement in people s lives from our investment. What is VFM analysis? VFM analysis is the use of various measures and metrics to analyse programme cost, process and performance data to assess factors of economy, efficiency, effectiveness, and equity - the four E s of VFM. In practice, planning and assessing VFM involves analysing programme data that, taken together, tell the story of how inputs have been translated into outputs, outcomes, and eventual impact. More will be detailed later about the types of data used in VFM analysis, but it is common that project plans, logframes and work plans, budgets and financial data, performance targets and actual performance, and beneficiary data are all part of a basic VFM analysis. Common VFM measures and metrics include but are not limited to: cost-to-budget and cost-to-performance ratios; unit-costing; disaggregated beneficiary analysis; cost-efficiency and cost effectiveness assessment; cost-benefit analysis; procurement review and spot-checking; human resource trending; and various projectspecific outcome and impact measures. VFM analysis uses an array of analytical tools that enable a) types of measurements related to the cost and performance of development projects/programmes; b) comparison of measurements across sites, regions, implementing partners and intervention types; c) improved performance management by understanding the factors that cause cost and performance variances from programme expectations; and d) regular VFM reporting to programme management and to donors to ensure that the greatest results possible are being gained from investments in development projects/programmes. The intent of VFM analysis of programme data is to discover whether and how inputs (money and other resources) have been translated into the desired outputs, and to determine whether that process has been economic, efficient, effective, and equitable, and to use the information developed by VFM analysis to make programme improvements. Conclusions are then derived from the analysis of data about the economy, efficiency, effectiveness, and equity of programme operations. VFM process, factors and measurements There are four factors that are measured in VFM analysis as described in Table 1. Value for Money (VfM) Process Notes P a g e 4
Table 1 VFM Process Factors VFM Factor Description Indicators Possible Measures Economy Efficiency Effectiveness Equity Source: Makana Consulting, LLC Related to the all-in cost of purchased inputs (goods and services) which are then translated through programme processes into beneficiary outputs, outcome, and impact. Related to how well the inputs are translated to outputs. Related to sustainability and permanence of an output; related to the effect of an output in generating and sustaining an outcome. Related to measuring the spread of programme benefits to marginalized and poorest target populations. May vary somewhat by programme focus. Assess if procurement practice meets international best practice; assess procurement limitations due to context; assess procurement timeliness; consistent supply indicates proper procurement planning; review competitive bidding; review of transport, warehousing and other costs. Performance review: actual to targets analysis; cost review: actual to budget; cross-site performance to cost review; IP cost to performance review. Estimated or actual number of beneficiaries who experience the desired outcomes. HH or other surveys that capture indicators of sustained use, functionality, and behaviour change over time. Disaggregated benefit analysis by target group; measures of disaggregated groups receiving benefits. May require field verification. Procurement planning review; unit-costs; comparative costs across sites; procurement transaction costs (time, post FOB (Free on Board, or when the recipient takes ownership from the shipping agency) warehousing and distribution, etc.); HR staffing review; trend specific costs over life of programme. Cost-to budget and performance to target ratios; cost-performance ratio; output unit-cost per beneficiary. Changes in diseasespecific morbidity and mortality (long term) 5 ; year to year income changes; education attainment measures (Note: verified benefits/outcomes measured via Household (HH) and village surveys or other means are preferable to estimated benefits/outcomes). Analysis by multiple factors of disaggregation: gender, age, ethnicity, wealth quintile, etc. The above four factors are primary to VFM analysis. As the description indicates, multiple data sources and analyses are necessary. VFM process descriptions Figure 1 is a simplified graphic flow-chart showing the process of translating money into outputs, outcomes and impact in a typical development intervention. It represents DFID s common understanding of the VFM process. The stages in the process are linked to the VFM factors of Economy, Efficiency, and Effectiveness. Equity may be measured at different stages in the process. Below the VFM flow chart are suggestions of where 5 Long term benefits may be measured after a programme ends. Value for Money (VfM) Process Notes P a g e 5
in the process certain high-level measures may be undertaken. Of course, measures of Economy, Efficiency, Effectiveness, and Equity may be taken at various points in the process of a development intervention. The art of VFM analysis is knowing what to measure, when to measure, and which measures will most accurately reflect actual programme performance. Figure 1 VFM Process Chart Source: Makana Consulting, LLC. Economy: Efficiency: Effectiveness: Equity: Were the highest quality inputs purchased, and were they purchased at the best price? How well were the inputs converted to the desired outputs and transferred to the beneficiaries? Did the outputs achieve the desired outcomes and impact? Were these inputs, outputs, and outcomes equitably distributed among vulnerable groups of beneficiaries? (Vulnerability may be defined by gender, region, ethnicity, age, or other relevant categories.) How VFM analysis is used There are several uses for VFM analysis: Commonly VFM analysis is part of the mid-term or end of project evaluation. DFID Business Cases for new investments require an abbreviated VFM analysis of options. VFM analysis is used to assess the value of applications to grant-making facilities in development programmes. VFM is used to monitor the activity of Implementing Partners to compare performance and cost across Partners and to monitor and strengthen performance. VFM is used to measure the cost and results of similar interventions in different regions/sites of a country programme. When accompanied by narrative that details differences between countries or regions, VFM is used to understand the costs of similar interventions in different contexts. In each example above, VFM analysis leads to conclusions about performance, costs, the economy and efficiency of operations, and possible measures of effectiveness. Value for Money (VfM) Process Notes P a g e 6
What is needed for a VFM analysis? Data. Each programme needs to establish for itself the data array that will enable value measurements that address the VFM factors (the Four E s). Fundamental for any VFM analysis is the logframe or results framework that has been agreed upon between the donor and the programme implementing agency (ies). Performance data for each results Indicator and for each activity are also needed. In addition, expenditure to budget data that are aligned to the performance data are also required. Beneficiary data are also required, and these may be estimated 6 and/or actual. Beyond these fundamental data, most programmes work with one of more implementing partners, and the same cost and performance data are needed from each Implementing Partner (IP). Procurement data, human resources staffing data, and household/community/clinic or other outlet survey data are often useful. The relevant data to show metrics of VFM (economy, efficiency, effectiveness, and equity) will vary with each programme analysis. Challenges to VFM analysis The primary challenge to robust VFM analysis is the availability of data. The second major challenge is that the available data are seldom aligned to allow easy comparison of performance and cost. Data availability is a persistent challenge. Programme performance and finance data are often separated in departments that seldom communicate about performance. Sometimes programme data are stored in-country and finance data are stored and analysed at a distant global office, making communication even more difficult. Finance and programme staff communicate infrequently, programme staff are sometimes math-phobic, and finance staff may have limited knowledge of the practical issues and challenges facing programme staff in country. So, the first challenge is simply acquiring the right data from the right sources. A second challenge is aligning cost and performance data for analysis. Most often, performance reporting is activity-and-output defined, often using the logframe as the primary M&E reporting framework. Reporting is at the activity and sub-activity level, sometimes cumulative to the output level. Conversely, finance reporting is seldom coded to the programme activity level. Thus, to align finance and performance data it is often necessary to expend considerable time simply reassigning expenditures from different budget categories to particular activities and outputs. This is a lengthy process and is subject to possible error and inconsistency. IP s may be reluctant to share financial and/or performance data for fear of losing competitive advantage by sharing business information. Government agencies often are reluctant to share data, and many times government data are inconsistent with data from other independent sources. There is no fully satisfactory solution to these challenges. Each must be addressed and worked-through; it is time-consuming; and inevitable compromises will be made in the type, focus, and breadth of VFM analysis and conclusions. During a VFM analysis, it is essential to enumerate the data challenges that were encountered and to indicate if proxy data or other work-arounds were considered to determine the ultimate measurements. 6 See later sections for the limitations of estimated beneficiary data. Value for Money (VfM) Process Notes P a g e 7
CONDUCTING VFM ANALYSIS Approaches and tools for programme managers Step 1: Establish your analytical scope You cannot measure everything and do not need to do so. Decide what to monitor and measure. Asking simple questions will help. Overarching Performance Questions: What was the assessment plan at inception? Was VFM part of the inception plan? Do the data capture methods support the granularity in performance measurement needed for effective VFM assessment? Are expenditures coded to link expenditures to identified logframe activities and outputs? How do you currently measure the programme performance? What are your key performance indicators? What must you report on to donors? What are the key performance indicators that you monitor for implementing partners? What are your key logframe indicators? Do you have performance data at the activity level or only at the output level? Overarching Finance Questions: Are your finance data aligned to match performance data? In other words, do you have an activity-based budget? If not, which is likely, how can you match the planned budget and the actual costs for key indicators above? How can you access the financial data for each output and for each activity? How can you capture the data for IP s? Once you answer the finance and performance questions you should be able to create a data framework using the steps below. Step 2: Set up programme data to use VFM analysis Whether you are using VFM as an internal performance management tool, are proactively undertaking a VFM assessment for your organization or for a donor, or are preparing for an external VFM review, these simple steps are essential. Measuring Performance 1. Identify and summarise the features of the programme; its context, origins, challenges, and gains made to date; and key programme strategies. It is useful at this stage to refer to the DFID Business Case and Inception documents to provide context to the donor s initial plans and strategies. 2. Decide what you want or need to measure; in other words, what will be the scope of your VFM analysis? a. Begin with the programme s inception documents and the logframe and/or results framework: i. Programme progress is measured by comparing planned output milestones vs. actual results, per time period. Be aware that some outputs may be more important than others and can be weighted accordingly. Create a separate spreadsheet for these data. ii. Programme activities that contribute to, but are subordinate to, outputs are usually measured. Measurement would be similar to the above output measurement: activity milestones or targets vs. actual results. You would normally want to analyse at this level to understand how an activity met its targets as part of understanding how the activity 7 contributed to the overall output progress. Add the data to the spreadsheet. 7 An activity, when referring to logframe performance, is a programme intervention that supports the achievement of a larger output. An example would be household or community level sanitation marketing that supports the Output of increased numbers of communities being declared ODF. Value for Money (VfM) Process Notes P a g e 8
iii. iv. Outputs such as governance change or capacity-building are difficult to measure; such changes are often difficult to quantify and are not always captured by logframes and results frameworks. Nonetheless, such changes can have enormous long-term impact and can improve the value of donor inputs. Useful measures of qualitative change often reflect the context. For example, local government support to CLTS, or support to poorest households might indicate positive governance change especially if sustained over time. Local government support for Operations and Maintenance funds for water points, combined with user contributions, might indicate greater community-local government cooperation for improved water source functionality and sustainability. It is challenging to measure governance change, but there is room in a VFM assessment for qualitative improvements, especially if qualitative changes are backed by robust evidence over time. The actual governance or capacity-building indicators to be measured are contextual, ideally show improvement over time, and can be measured repeatedly. An important goal or ambition for VFM assessments is to measure the process of input translation to outputs during a project, and to measure post-programme outcomes and impact after programme end. Ideally, a VFM assessment will measure outcomes and impact. To do so often requires a VFM plan that begins at inception and continues with VFM measurements after the end of a programme, to measure sustainable results over time. 1. Ideally, post-programme outcome/impact assessment should be built into programme planning and budgeting at inception. The data to be collected and analysed for VFM outcome assessment (effectiveness) will vary by programme focus (increased yield for agriculture interventions; reduced <5 stunting for nutrition interventions, etc.). 2. Data collection should be focussed on the actual, verified, quantifiable changes that have accrued to beneficiaries. v. Ideally, results measurement would include outcomes and impact. The obvious challenge to outcome measurement is that the evidence of improved outcomes is often apparent after the end of a programme or project. Outcome measurement for projects or programmes of three years or less duration is unlikely. For longer programmes, outcome measurement including HH surveys or other beneficiary measurements are best included as part of programme inception planning. It is important to quantify and or monetize benefits when possible, and set up data collection methods at the programme outset so that the necessary data is gathered during programme operations to feed into a post-programme outcome assessment for VFM analysis. vi. If you cannot plan for outcome or impact analysis, develop an analysis framework that includes all outputs, activities, and sub-activities for which you have performance data. You may not measure all activities, but set up your data framework to do so at the outset. It might look like this: Value for Money (VfM) Process Notes P a g e 9
Source: Makana Consulting, LLC. Measuring Cost Table 2 Performance Data Alignment Format Intermediate result Indicator 2013 2014 Milestone Actual Perf. Ratio Milestone Actual Perf. Ratio Output 2 # of villages declared ODF Number of Municipal Wards declared and Output 2.1 (IR 2.3) Service Delivery certified ODF status Output 2.1 (IR 2.3) Service Delivery Number of public toilets constructed Output 2.1 (IR 2.3) Service Delivery Number of people benefiting from WSP only Number of people benefiting from schemes Output 2.1 (IR 2.3) Service Delivery with WSP and improved functionality Output 2.2 (IR 2.5) Child-friendly basic and secondary education Number of schools/child clubs with WASH software (one star) Output 2.2 (IR 2.5) Child-friendly basic and secondary education Number of schools/child clubs with WASH software (two star) Output 2.2 (IR 2.5) Child-friendly basic and secondary education Number of schools/child clubs with WASH software (three star) Output 2.2 (IR 2.5) Child-friendly basic and secondary education Number of people trained WASH in schools (formal training) Capacity Building WASH People trained - women Capacity Building WASH People trained - men 1. The next step is to compare actual costs (expenditures) vs. budget per each output for the same time period as the progress reporting above. You need to decide what costs to include and the question is often raised about measuring direct vs. indirect costs. 2. Defining Direct and Indirect Costs: a. Direct costs are those for activities or services that benefit specific projects, e.g., salaries for project staff and materials required for a particular project. Because these activities are easily traced to projects, their costs are usually charged to projects on an item-by-item basis. b. Indirect costs are those for activities or services that benefit more than one project. Their precise benefits to a specific project are often difficult or impossible to trace. For example, it may be difficult to determine precisely how the activities of the Director of an organisation benefit a specific project. 3. It is possible to justify the handling of almost any kind of cost as either direct or indirect. Labour costs, for example, can be indirect, as in the case of maintenance personnel and executive officers; or they can be direct, as in the case of project staff members. Similarly, materials such as miscellaneous supplies purchased in bulk pencils, pens, paper are typically handled as indirect costs, while materials required for specific projects are charged as direct costs. a. Direct costs only: for some programmes it makes sense to only compare direct costs, that is, the costs directly associated with a particular intervention or milestone. Reasons to only compare direct costs include: you have comparative data from other programmes and sources that use direct costs only; you are comparing costs across multiple sites in the same programme and direct costs to deliver an intervention allows you to be more precise in comparative performance costs; you want to compare the costs difference between easy and difficult to reach areas and direct costs will show that variation more clearly; indirect programme support costs are so difficult to estimate/calculate that you believe using indirect project costs will make the cost data less credible. Despite limitations, calculating direct costs only, as was done with the Nepal Case Study, is a credible approach to assess VFM and management performance. b. Direct and indirect costs: for some programmes it is possible to credibly estimate/calculate the direct and indirect costs of an intervention. In this case it is valuable to analyse the total all-in costs of an intervention as you will capture changes in programme staffing numbers, intensity, and costs during the life of a programme. With this method it is often possible to capture improvements in efficiency as the programme develops over time, an indicator of VFM. c. While the measurement of indirect costs is not always necessary indirect costs may be valuable to assess when a programme is implemented by multiple partners, in multiple regions, and in which unit costs vary greatly. Indirect cost measurement can provide a window into internal operations costs that may contribute to cost variances. It is necessary to be certain that the categories of indirect costs are consistent. 4. Whichever method you choose, it is important to state clearly whether you are measuring direct or indirect costs, or both. Within a programme, use the same cost data for all interventions (do not mix Value for Money (VfM) Process Notes P a g e 10
direct-only costs of one intervention and direct and indirect costs for another intervention in the same programme). a. As noted, aligning cost data with performance data is challenging, often due to the lack of activitybased budgeting by programmes, along with expenditure coding categories that do not clearly relate to logframe or results framework activities and outputs. For instance, behaviour change social-marketing might be reflected as a single activity in a logframe with clear results targets, while expenditures for this activity may be split across categories of staff, transportation, equipment, training, venue hire, etc. Conducting robust VFM analysis requires capturing all the costs related to an activity/output and calculating results to total cost. i. There is no simple solution to this challenge. Foresight and pre-planning for VFM measurement at regular programme intervals will necessarily require communication between programme management, M&E, and Finance staff to enable mapping for consistent capture of costs related to results. b. The first way to compare costs is by inputting the approved budget figures, the actual figures, and calculate a percentage of spend to budget. It might look like this: Source: Makana Consulting, LLC Table 3 Finance Data Alignment Sample Intermediate result Indicator 2013 2014 Budget Spend Sprend Ratio Budget Spend Spend Ratio Output 2 # of villages declared ODF Number of Municipal Wards declared and Output 2.1 (IR 2.3) Service Delivery certified ODF status Output 2.1 (IR 2.3) Service Delivery Number of public toilets constructed Output 2.1 (IR 2.3) Service Delivery Number of people benefiting from WSP only Number of people benefiting from schemes Output 2.1 (IR 2.3) Service Delivery with WSP and improved functionality Output 2.2 (IR 2.5) Child-friendly basic and secondary education Number of schools/child clubs with WASH software (one star) Output 2.2 (IR 2.5) Child-friendly basic and secondary education Number of schools/child clubs with WASH software (two star) Output 2.2 (IR 2.5) Child-friendly basic and secondary education Number of schools/child clubs with WASH software (three star) Output 2.2 (IR 2.5) Child-friendly basic and secondary education Number of people trained WASH in schools (formal training) Capacity Building WASH People trained - women Capacity Building WASH People trained - men 5. As you develop performance and cost spreadsheets, keep in mind that you are preparing to compare performance and cost for each output, activity, or sub-activity. How you set up your spreadsheets and workbook is a choice. Some prefer to have separate worksheets for financial data and to create the comparisons through macro-enabled sheets. Others prefer to align the financial and performance data in one worksheet and present the comparison in one sheet. a. Either method permits presentation of the raw data and charts and graphs for data visualization. b. To measure activity inputs (expenditures) it is often more useful to group certain costs that are still more granular than simple measuring costs for Outputs, but which enable measurement of key activity performance. For example, grouping CLTS software costs, distinct from CLTS hardware, would effectively separate very different activities without measuring the details of costs for training, curriculum, etc. Step 3: Display your cost and performance data Spreadsheets are not always user-friendly and, as a project develops, the amount of data in a spreadsheet can be overwhelming. Data visualization methods like charts and graphs can help communicate statistical information clearly and efficiently. These methods are also critical first steps in the data analytic process. For instance, Tables 4, 5 and 6 below presents data in three different forms: Value for Money (VfM) Process Notes P a g e 11
2013 2014 2015 2016 2017 DFID Accelerated Sanitation and Water for All Independent Process Monitoring and Evaluation Table 4 Performance data and data visualization Number of Districts with complete WSP 2013 2014 2015 2016 2017 Target 20 20 20 20 20 Actual 12 16 - - - Ratio 60% 80% 0% 0% 0% 25 20 15 10 5 - Table 5 Bar Chart Data View 16 12 Actual Target - - - 2013 2014 2015 2016 2017 120 100 80 60 40 20 Table 6 Trend Chart Data View - 20 40 12 60 28 80 100 Target Actual Linear (Actual) 1. The bar chart in Table 5 presents annual data and targets in a simple graphic. Table 6, a trend chart presents the same data but with additional nuance. Whereas in Table 5, while not meeting targets, a project manager might be inclined to view the progress from 2013-2014 as an indication that 2015 will be even better performing. Table 6, the trend chart, presents the data in a more informative fashion. Here, even though there was good progress from 2013 to 2014, it is clear that even the rate of progress in 2014 must improve further to meet long term project goals. 2. It would be possible and useful to create a third table, to accompany tables 5 and 6, showing what progress is needed each year to catch up and meet targets by the End of Programme (EOP). In other words, visualising the data is another way to help manage programme performance. 3. Cost and performance data can be displayed multiple ways, in separate charts for cost and performance or in combined scatter charts. The choice is yours; the display of data should be carefully chosen to display the project data in ways that provide clear evidence for programme management decisionmaking. Step 4: Compare costs and performance There is growing interest in cross-country cost-to-performance comparisons of development interventions. It is both useful and potentially misleading to compare costs across multiple sites or countries for similar interventions. Addressing the potential to mislead first, comparing costs across sites or countries usually reduces the complexity of a programme in different contexts to a single number. The unit-cost for access to a water point may be vastly different in an urban area versus a well-developed small town versus a rural mountainous village. It is misleading to present comparative unit-costs in different regions without substantial supporting data and narrative to explain and justify different costs across different locales. Comparative unit-costing should be undertaken cautiously. Value for Money (VfM) Process Notes P a g e 12
With that caveat, in-country comparative unit-costing, in particular, is a valuable performance management tool. Unit-cost-per-output comparisons across different regions in the same programme can be informative. Programme management should know why costs vary between sites for the same intervention. Perhaps there are logistical or other factors that drive costs differently. On the other hand, perhaps one site is over-staffed compared to other sites thus contributing to higher unit-costs. Or perhaps procurement delays have created supply limitations in one site, causing lost days of programme productivity and lower output volumes, and thus higher unit costs. Used carefully, cross-site unit-cost comparisons can provide programme leadership with evidence to target performance management support. Step 5: Determining estimated or actual beneficiaries; disaggregating beneficiaries Beneficiary analysis often disaggregates beneficiaries by groupings that are useful to the programme, usually particular target groups. There are multiple ways to disaggregate beneficiaries; common disaggregation includes: by gender, age, rural or urban, wealth quintile, locality or region, and ethnic or tribal identification. The discussion below can be adapted to any particular beneficiary grouping. Who are Beneficiaries? 1. It might seem obvious that programme beneficiaries are the recipients of programme goods and services. It is not that simple. Beneficiary designation needs to be carefully determined in advance of calculating beneficiary benefits. Some examples illustrate the complexity of designating beneficiaries. a. Are beneficiary numbers planned or anticipated, or actual and verified? It is common for programme design to estimate the number of anticipated beneficiaries, often using well-designed factors to estimate numbers. But there is almost always variance between even well-designed estimates and actual beneficiaries. The difference in the number of beneficiaries counted creates different unit-costs. Often the data leave you no choice but to use estimated beneficiary numbers. Determine what your available data allow you to count credibly, and then indicate whether the beneficiaries are estimated or actual. b. Estimated vs. actual beneficiary numbers may vary because: i. Initial beneficiary estimates were incorrect. ii. Cultural, social, and political issues may limit some users from receiving or using the benefits. iii. Societal pressures may discourage beneficiaries from reporting use of a programme benefit. iv. More or fewer beneficiaries may use the benefit than planned. Sometimes people travel great distances to use a benefit in one place that is not available elsewhere. c. Beneficiary access is different than beneficiary use/serviced uptake. For example: i. Twenty water points were developed in a village designed to serve 300 people each. Thus, a total of 6,000 people can be estimated to have access to water through the programme. Thus, there are 6,000 estimated beneficiaries, and a unit-cost derived from estimated beneficiary numbers would also be estimated. Both of these are clearly subject to error. a) Some VFM analysts would term the potential use of a good or service (such as access to a water point) to be an intermediate result because potential rather than actual users are being measured. ii. If a village survey of use of the water points was undertaken, quantification of actual water point beneficiaries would be possible and would be more accurate than estimated beneficiary numbers. In the case of village water points, a village survey might indicate some water points were not preferred due to location, and others may be dominated by a few families. In each instance, exogenous factors limit the actual uptake of water access, reducing beneficiary numbers. Unit-costs using actual user data generate more accurate unit-cost calculations and also provide evidence to managers to further manage performance. Value for Money (VfM) Process Notes P a g e 13
Step 6: Calculate unit-costs Unit-costing is of particular interest to some programmes and donors. Unit-costing can be calculated differently depending upon need. For example, unit-cost could be derived simply by dividing the total cost of a good or service by the number of anticipated beneficiaries over a specified time period. Or unit-cost could be derived by dividing total cost by actual beneficiaries, as verified by a combination of means that may include surveys, Multiple Indicator Cluster Survey (MICS) data, community reporting structures, etc. Since anticipated and actual beneficiary numbers will vary, unit-costs will differ depending on the method of determining beneficiaries. Unit-costs using verified beneficiary data generate more accurate unit-cost calculations and also provide evidence to programme managers about beneficiary uptake of programme service so that they can further manage performance. Increased user uptake decreases unit-cost and increases VFM (economy and efficiency). Small changes in programme delivery that increase beneficiary uptake of services have a profound impact to increase VFM. Using cost, performance and beneficiary data to determine unit-costs 1. Approaches to unit-costing can vary a. Table 7 illustrates ways to measure unit costs. i. You must first determine if your beneficiaries are estimated based on a programme plan. In this example, the plan is to cover 10 villages averaging 200 individuals, so the planned beneficiaries are 2,000. If 9 villages are covered, the estimated beneficiaries would be 1,800. This is a rough beneficiary estimate. b. Unit-cost can be further refined by narrowing the focus from planned to actual beneficiaries. This requires field verification of actual beneficiaries and generally results in more accurate unit-costs. i. In the same 10 villages you may find that the actual population is 2,200, not the 2,000 estimated beneficiaries. You may also find that the actual coverage across villages averages just 85%. ii. iii. iv. Thus, if 9 villages are covered with the higher actual population averaging 2,200, the possible beneficiaries would now be 1980. Since you have verified that coverage is only 85%, the total actual beneficiaries is 1,683. Similarly, verification of functionality also affects actual beneficiaries; a decrease in functionality renders a decrease in actual beneficiaries, resulting in higher unit costs over time. Table 7 below shows how unit-costs vary depending upon the data provided. In this example costs are unchanged and the variable is how beneficiaries are measured. Table 8 shows cross-site variations in the same programme. v. Table 7 shows that lower than anticipated actual beneficiaries increases unit-costs. The obvious implication of such data is that programme management can lower unit-costs and improve the VFM of the intervention by determining if increased adoption of an intervention (increased actual beneficiaries) can be achieved by further focused activity (increased social marketing, improved training, etc.) By following up with targeted support to increase the adoption of an intervention, unit-cost data provides evidence upon which programme managers can make decisions to improve performance. Value for Money (VfM) Process Notes P a g e 14
Table 7 Calculating Unit-costs-Variability Beneficiary Category Estimated beneficiaries by village and estimated unit cost Actual population by village and estimated unit cost Actual average beneficiaries by village and actual unit cost Planned cost per village Actual cost per village Beneficiary totals 2,000 2,200 1,683 10 villages 9 village Costs 50,000 50,000 50,000 50,000 50,000 Unit-cost 25.00 22.72 29.70 5,000 5,555.55 Table 8 illustrates performance variability across sites before and after a malaria net campaign and related social marketing. The next step in VFM analysis would be to link the performance variability across sites with costs for each site activity. It would then be possible to conduct a cost-to-performance analysis for each site, develop unit costs and if variance of unit costs is evident, investigate the factors causing varied unit costs. Similarly, using the actual unit-costs of interventions, rather than the higher estimate unit costs, for future planning to enable greater coverage for the same costs. Using data to manage performance, as in Table 8, increases the efficiency of programme interventions. Table 8 Calculating Performance Across Sites Indicator Site A Site B Site C Site D Before Campaign % HH w/ any net % HH w/ any Insecticide treated net 13.2% 0.7% 4.6% 7.6% 11.5% 0.5% 1.3% 2.0% After Campaign % HH w/ LLIN 69.3% 51.5% 52.5% 64.3% % LLIN used last night 68.4% 79.5% 53.3% 53.8% a. Unit-costs can be refined by target beneficiary group by measuring the uptake of services by gender, age, or socio-economic categories. This is especially useful to measure the variants in access and uptake of services by marginalized groups, and contributes to an equity measure of VFM. b. A key to useful unit-cost calculations is to get the most accurate data on costs and benefit uptake possible, as well as accurate cost data. When using the data it is necessary to identify the data source and to describe precisely how you define costs and beneficiaries. Setting up data for IP VFM Analysis and Performance Management Below is an example where a beneficiary analysis is linked to planned and actual costs and performance. The data are actual; the location and different IP s have been anonymized. Useful points about this data set up include: Baseline and Household (HH) data are actual; per HH cost is displayed as both planned and actual to date; performance is actual. Unit-costs are calculated both by estimation and by actual cost and beneficiaries. Value for Money (VfM) Process Notes P a g e 15
Costs and beneficiaries should be disaggregated by programme year. Columns H, I, and K reveal unit-cost outliers that could be informative for programme performance management. When data analysis reveals such variances-as often happens-it is a performance indicator that should be researched by programme managers to understand the factors leading to outlier costs. Programme management could use these data to further examine that factors that cause IP DEF to be the strongest performing IP for potential replication. Conversely, the lower cost-to-performance ratio by IP QRS demands management investigation to manage performance. Table 9 Measuring Performance, Cost, and Beneficiaries IP s a b c d e f g h i j k Baseline data (HH) Baseline population Estimate Per HH cost Present coverage (%) Present coverag e (HH) Current VDC unitcost Current HH Unitcost Curren t per capita unitcost Expected target beneficiary by end of FY HH Unitcost expected by end of FY ABC 17,237 112,202 435.11 25.91 4,466 625,000 1,679.32 66.84 7,400 1,013.50 DEF 4,566 26,247 1,007.45 20.72 946 920,000 4,862.19 175.26 2,739 1,679.40 HIJ 13,277 87,074 195.83 25.31 3,360 162,500 773.72 29.86 6,655 390.66 KLM 6,042 38,906 662.03 17.50 1,057 666,667 3,783.04 102.81 3,044 1,313.91 NOP 33,402 203,693 287.41 42.11 14,066 342,857 682.52 47.13 19,748 486.14 QRS 31,800 217,937 471.70 16.48 5,241 500,000 2,862.25 68.83 9,396 1,596.49 Total 106,324 686,059 29,136 531,845 485.21 75.20 Value for Money (VfM) Process Notes P a g e 16
Using VFM measures to understand efficient and economic options VFM analysis can assist programmes choose the most economic, efficient and/or cost-effective option when multiple intervention options exist. Table 10 uses verified data from a malaria eradication campaign. Different options were studied: a) provision of Long-Lasting Insecticidal Nets (LLIN s) through a health awareness campaign, b) net distribution accompanying routine ante-natal care by community health workers, and c) distributing nets through private sector outlets. Table 10 Generating Increased VFM by Understanding the Benefits of Different Options Distribution Channel Volume Distributed Avg. Cost per LLIN Extrapolated Night-before Use Use-cost per LLIN Use-cost if+ 10% use Campaign 36,149,785 3.66 63.75% 4.97 4.62 Routine ANC 998,082 3.79 63.75% 5.16 4.78 LLIN Commercial Sector 612,769 2.12 63.75% 2.89 2.68 In actual practice, all three channels were used. The weakness in this data is that the most important variable remains estimated: Night-before Use of a distributed net. Access to or possession of a net did not bring about the desired beneficiary benefit unless it was used. Thus data to determine actual use would generate more credible unit cost data; likewise, programme interventions to increase use would increase VFM by decreasing unit costs. And, of course, increased use of nets was assumed by the programme to lead to desired programme outcomes and impact. As noted previously, verified beneficiary data is essential to targeting inputs for effective interventions. In this case it is clear that commercial sector engagement dramatically reduces the unit cost of an intervention, though its scale and scope is much more limited than a broad-scale LLIN campaign. It is valuable to note that VFM analysis is not intended to drive all interventions to the lowest cost point but rather, to understand the performance and cost implications of various strategies. As above, it seems evident that a broad net distribution campaign would achieve rapid and broad distribution; continued supply of nets through routine ANC outlets would potentially protect the vulnerable <5 population, and the commercial sector will supplement the above by targeting that population seeking private vendor medical treatment. Over time, the VFM analysis could quantify the cost-effectiveness of broad scale distribution vs. private-sector distribution to targeted populations. Procurement analysis 1. As part of the Economy analysis for VFM, a procurement review should be undertaken for any programme or project intervention in which the purchase of goods and commodities is a major cost driver of programme expenses. Examples of programmes that may require a procurement review include: a. Health and nutrition programmes in which drugs, nutrition support commodities, and/or clinic supplies are key components; agriculture programmes that provide inputs to project beneficiaries either directly or indirectly; education programmes supplying learning supplies and textbooks; and programmes supplying health prevention supplies (malaria nets, condoms, etc.) b. Training programmes relying on newly printed materials or frequent facility rental. c. Infrastructure development programmes. d. Programmes making extensive use of consultants. e. Not all programmes require a procurement review. Examples of programmes that may not require a procurement review include governance legal and public finance reform programmes that rely upon a small number of well-placed staff to work on policy issues. Value for Money (VfM) Process Notes P a g e 17