Sustainability & Cost-Effectiveness Assessment Guide

Transcription

1 Performance Measurement Framework for Business Development Services USAID Microenterprise Best Practices Project Field Research (Development Alternatives, Inc and the SEEP Network) April 9, 2001 Sustainability & Cost-Effectiveness Assessment Guide Mary McVay April, 2001

2 TABLE OF CONTENTS Introduction... 3 The Goals of Assessing Sustainability and Cost-Effectiveness... 3 Research Questions... 4 Using this Guide... 5 Indicators AND RATIONAL:... 5 methodology Sustainability Ratios Methodology: Cost-Effectiveness Ratios Data presentation, analysis & reporting... 21

3 INTRODUCTION This guide to assessing sustainability and cost-effectiveness in BDS programs is part of a global initiative to explore valid, practical, and useful performance indicators for Business Development Services (BDS) programs. It is designed to be used in conjunction with the PMF Field Research Guides on Impact and Market Development Assessment. The indicators have been selected from Best Practices studies and consensus-building global conferences. They reflect common goals of most BDS programs: increasing outreach and reaching large numbers of people through the development of vibrant, private sector BDS markets, delivering services through sustainable suppliers and cost-effective programs, and having a positive impact on client firms. The focus of this guide is the assessment of sustainable BDS suppliers and assessing the cost-effectiveness of BDS programs. The set of indicators is proposed here for testing by six BDS programs as part of global Field Research (funded by USAID through the Microenterprise Best Practice Project, managed by Development Alternatives, Inc. and the SEEP Network). It is expected that the Field Research will yield findings that will modify the guide substantially, and propose avenues for further research. The Performance Measurement Framework (PMF) is not intended as a holistic BDS program evaluation system. Rather, it is a snapshot of performance indicators that could be used to provide program managers with useful data to improve services to small enterprises (SEs), to compare programs in order to identify best practices, and to assist in establishing program funding and performance guidelines. The Goals of Assessing Sustainability and Cost-Effectiveness The broad goals of sustainability and cost-effectiveness focus on the performance of the institutions that supply or provide BDS to micro, small and medium sized firms (SEs). The two goals are related, but substantially different. In the context of the PMF, here are working definitions: Sustainability: the ability of an institution to offer services over the long term; the financial profitability of a BDS supplier; and/or the financial viability of a particular BDS. The terms cost-recovery, sustainability and profitability measure the same thing, but express different results. In other words, one program may cover only 40% of its costs, another may achieve financial sustainability, and still another may generate profits - but one can use the same ratio of expenses to revenues to measure these different outcomes. Cost-Effectiveness: program expenses compared to program results or outcomes, such as number of people served, number of people who benefited from the service, etc. The PMF set of cost-effectiveness indicators also includes a simplified cost-benefit ratio. Cost-benefit looks at program costs compared to the impacts of the program, for example the profits generated in client businesses as a result of the service. Sustainability focuses on whether a service will be available to businesses in the long run. The assumption is that financial sustainability is the best indicator of whether an

4 institution will be able to offer services over the long term. In addition, by assessing financial sustainability, the framework attempts to assess the subsidy level of the service and, by inference, its potential for crowding other services out of the market. Finally, it is assumed that most suppliers, particularly private-sectors suppliers, are concerned with financial sustainability or profitability. This framework looks at two types of sustainability: 1) the sustainability of the services itself: to what extent are SEs able and willing to pay for the cost of the service? How profitable is the service? And 2) the sustainability of the BDS supplier: to what extent is the supplier institution (whether forprofit or non-government organizations (ngo) profitable or able to cover its costs? For many BDS practitioners and donors, sustainability is a sufficient indicator of success. If SEs are willing to pay for services and they can be supplied profitably, then the services have become part of the market, and that is success. However, many practitioners and donors are equally concerned about the impact of services, and whether the impact is being delivered for a reasonable cost. The PMF attempts to address this question with its cost-effectiveness indicators. This set of indicators considers total program costs, and compares them to program outcomes and benefits, although it is by no means a thorough cost-benefit analysis. The cost-effectiveness indicators proposed here are simplified indicators that may be appropriate proxy indicators for cost-benefit analysis, which is generally too complex for most practitioners to carry out. Increasingly, there are institutional separations between BDS suppliers - organizations that directly serve SE and hope to become sustainable - and "facilitators" - non-profit organizations that support suppliers by providing training, market research, and other services. In programs with this structure, "sustainability" is usually a goal of the supplier, but not the "facilitator." Cost-effectiveness analyses the facilitator's financial performance. The application and interpretation of both sets of indicators depends on the application and interpretation of other parts of the PMF. The financial sustainability indicator is intended to be interpreted in the context of the other performance indicators, particularly program scale and impact. The Cost-effectiveness indicator is dependant on the collection of impact data and is thus also to be interpreted in the broader context of the PMF. These indicators contribute to the PMF an understanding of the institutional performance of the BDS organizations involved in supplying the service: is the service and the supplier financially sustainable? What was the cost of the program, relative to the results achieved? Research Questions The PMF Field Research is examining particular issues in each performance category as the Field Research Team tests the application of these indicators. In general, the PMF Field research will examine whether these indicators are valid and practical, how they may be interpreted and applied, and whether or not they lend themselves to fair comparisons of program performance. With regard to Sustainability and Cost- Effectiveness in particular, the following are the key research questions the Field Research Team will explore:

5 Sustainability: 1) How easy is it for BDS programs to allocate operational costs and revenues to specific services, programs, or SME clients? How standard and fair can these allocation categories be? 2) Is cost recovery a good proxy indicator for impact? Cost-Effectiveness: 1) How easy is it for BDS programs to allocate cumulative program costs, particularly overheads, to specific services, programs, or SME clients? How standard and fair can these allocation categories be? 2) How valid is the simplified cost-effectiveness analysis based on the impact data collected with PMF methodology? 3) How valid and useful are the more simple cost-effective indicators: cost per firm assisted and cost per supplier assisted? Thus, the major area of concern for both these indicators is cost allocation, and how the indicators may be interpreted and applied. Using this Guide The authors suggest that this guide be used in conjunction with the other guides for PMF Field Research. Definitions for the other guides apply here unless otherwise noted. A thorough read through the guide before beginning will probably be useful. The final section contains the tables and formats for reporting data. The attached spreadsheet (PMF Sust & CE Tool is an optional tool that may be useful in helping with calculations and data presentation. The guide first presents the indicators, their definitions, basic methodology and interpretation, ad the rational for selecting these indicators. Then, it presents the detailed methodologies to be used in calculating and interpreting the indicators. Finally, the guide suggests tables, content and format for the report. INDICATORS AND RATIONAL: The following two tables summarize the sustainability and cost-effectiveness indicators, the corresponding methodologies that the PMF Field Research Team will be testing, and how the indicators should be interpreted. What are they telling us? Following the table is a detailed description of the methodology to be used in the PMF Field Research. All indicators rely on internal financial figures, financial data from BDS supplier partners, and from the PMF Field Research Impact and Market Development surveys. No new surveys need to be carried out, although some organizations may need to collect data from the BDS suppliers they support. Also, PMF field testers may need to work closely with their accounting departments to gather and report accurate data, and will need to use data from their impact surveys for the cost-effectiveness indicators.

6 Goal: Achieve supplier sustainability Indicator Proposed Methodologies Interpretation Financial Sustainability = Non-donor Revenues / Total Expenses BDS supplier financial sustainability BDS contribution margin BDS1 Contribution Margin = (SME Revenues from BDS1- BDS1 Direct Expenses)/ Total Expenses BDS2 Contribution Margin = (SME Revenues from BDS2- BDS2 Direct Expenses)/ Total Expenses Financial Sustainability ratio answers the question: last year, what percentage of total expenses of a BDS supplier were covered by non-grant revenue? The contribution margin of each BDS answers the question: Last year, what % of total expenses were covered by this particular BDS? When comparing different BDS, it answers the question, which BDS is more profitable for my institution? And so on. BDS viability BDS1 Viability = Revenues from BDS1/ Direct Expenses for BDS1 BDS1 Viability = Revenues from BDS2/ Direct Expenses for BDS2 This ratio tells us: How profitable was a particular BDS service? Last year, what percentage of the direct expenses to deliver a BDS were covered by the revenues generated for that BDS? And so on. Rational: As one advisor aptly put it: in a time when BDS practitioners are attempting to support private sector BDS markets, is best to use existing business ratios, rather than invent complex, donor-related ratios. So, rather than attempt to calculate a sustainability ratio that excludes specific donor-related costs, the organization-level indicator simply asks for the profitability of the institution - the bottom line, including all overheads, donor-related costs, depreciation, etc. 1 On the service level, we ask for 2 indicators. One is most helpful for BDS program managers: the contribution margin of a particular BDS. It helps assess the extent to which different services are helping to achieve sustainability or profitability. However, it is not very comparable across organizations, because it does not address to profitability of the service directly. The third ratio, which compares the expenses to the revenues of a particular service, reports the service profitability, excluding overheads and indirect costs. This ratio is helpful for looking at a BDS outside the context of its organizational home. Although it is difficult to standardize the method of allocating "direct" costs, when used in conjunction with the above indicators, which capture total costs, it should a useful indicator of whether a service is financially viable, while the other indicators reflect whether it is profitable within the institution that is offering it. 1 If you feel strongly that your true institutional sustainability is better reflected by calculating a sustainability ratio that excludes "donor-related costs" as you define them, you may ALSO report this ratio as you usually calculate it, providing details on what expenses are included and which are excluded. This is in addition to reporting the sustainability ratio described here.

7 Also noteworthy is which institutions should assess these sustainability ratios. There are generally 3 types of BDS providers or suppliers that may treat these indicators differently 2 : A. Supplier: An NGO or government agency that supplies BDS directly to SEs, and their main activity is supplying BDS to SEs. In this case, the sustainability ratios apply to the entire institution. B. Supplier Plus: An NGO or government agency that supplies BDS directly to SEs, and also supplies other services, to SEs or other clients. In this case, the sustainability ratios apply to the BDS program, and an appropriate allocation of overhead costs for the support provided by the NGO should be included in the calculation of total expenses. C. Facilitator: An NGO or government agency that supports other NGOs or private sector BDS suppliers. In this case, the sustainability ratios apply to the partners who are supplying the BDS. D. Facilitator/Supplier: Finally, many organizations do both - support BDS suppliers and serve SEs directly. In this situation, the sustainability ratios will apply to the main BDS organization and its partners. Separate calculations will apply for each institution. Finally, whether or not it is a goal of the program to become financially sustainable, it is important to assess financial performance because it reflects the extent to which services are subsidized and may be crowding other private sector suppliers out of the market. (The detailed instructions for gathering, calculating, reporting and analyzing sustainability ratios appear on page 10. Please take a moment to determine which description fits your program best, and note the corresponding title to enable you to better follow later instructions.) Below is a summary of the cost-effectiveness indicators. The proposed Cost-Effectiveness objectives and indicators are as follows. 3 Unlike sustainability, which is a private-sector oriented indicators, cost-effectiveness indicators are focused on analyzing the investment of public funds, and determining the value of benefits generated by this investment. They ask the question: Was the money spent to develop and deliver this service worthwhile to the businesses? Rational: Significant numbers of practitioners and BDS experts argue that sustainability alone is an insufficient indicator for program performance, that sustainability is not always a good indicator for impact. Rather, because there are so many weak BDS markets, in markets where there are few suppliers and few services available, entrepreneurs will pay high sums for services that may not, in the end, benefit their businesses. An example often cited is the large number of parents who send their children to technical schools even in labor markets where most graduates of technical schools do not find jobs. Thus, the PMF includes attempts at assessing impact, and, at analyzing impact compared to costs. Because it is meant to be practical, the types of 2 The PMF uses the term "supplier" to mean any institution that provides a BDS directly to an SE. Although it is essentially the same as a "provide," the PMF team prefers the more private-sector oriented term. 3 On the draft PMF that was agreed at the PMf virtual conference, there was a suggested provider income "total program cost per increase in provider revenue," that has been eliminated. For organizations that are a direct supplier or provider, the calculation is not complicated, but there is room for error, and there is little value added in the indicator. For example, is one year a program generated 20 cents for every dollar invested in program costs, and the next year only 5 cents, what does this really tell us?

8 impact indicators and the types of analysis must be simple, so a thorough, technical costbenefit analysis - comparing cost and benefits over time - had not been viewed as feasible for most BDS organizations. Instead, the PMF has devised a simplified costbenefit indicator and a series of cost-effectiveness indicators - indicators that compare annual program costs to annual program benefits and outcomes. It is hoped that, collected over time, these indicators will form a valid, practical and useful proxy for indepth cost-benefit analysis that could be performed on a more occasional basis. The indicators look at total program costs for the year most closely corresponding to the year that the BDS organizations conducted an impact assessment. Then, they compare these program costs to the different "impact" indicators: number of people acquiring, using and benefiting from a service, and the profit increase that businesses attribute to the service. As with the sustainability ratios, the procedure for calculating the ratios varies depending on the institutional set-up of the BDS program, so keep in mind what category your program falls into.

9 GOAL: Improve program cost-effectiveness Indicator Proposed Methodology Interpretation 1)Conduct an impact survey and identify the fiscal year most closely corresponding to the period of the impact survey. Calculate the annual program expenses for that fiscal year, for the BDS initiative for which you are tracking impact. 2)Calculate the ratio: Annual program expenses/ number of customers served that year Annual program expenses per customer served* Annual program expenses per supplier assisted* Annual Program Expenses per firm that "used" a BDS Annual Program Expenses per firm that reported "benefiting from" a BDS. Simplified costbenefit assessment comparing annual program costs to aggregate annual program benefits for entrepreneurs* Annual program costs/ number of suppliers served that year (if you are the only direct provider in your program, the number of suppliers is 1) 1) Estimate the number of firms that used a service that year by multiplying the percent of firms reporting having "used" a service, by the total number of firms assisted that year. 2) Then, divide the Annual Program Expenses by the estimated number of firms that used the service. (Expenses/#firms) 1) Estimate the number of firms that benefited from a service that year by multiplying the percent of firms reporting having "benefited from" a service, by the total number of firms assisted that year. 2) Then, divide the Annual Program Expenses by the estimated number of firms that used the service. (Expenses/#firms) 1)Estimate the total profits for firms assisted that year that may be attributable to the program. To do this, multiply the number of firms served that year by the % in the impact survey who attribute profit change to the service. Then multiply the average annual increase in per firm profits 4 by the estimated number of firms that would attribute change to the service, and add the results. 2)Calculate the ratio of aggregate annual benefits to annual program expenses. Benefits/cost. Express it in a ration and in percentage terms. This figure is the total program cost of delivering a service to one client. How much does it cost to serve one business? This figure is easy to calculate, but used alone would favor programs that serve many, but may not have a significant impact on the firms. This indicator illustrates the cost of assisting each supplier. It should be looked at in the context of the program's effectiveness in developing the BDS Market. This is an attempt to look at the relative cost of that market development strategy. This figure tells us the annual program cost for each client who applied the service in their business. It illustrates the cost of providing a useful service to a client. This number would normally be higher than the cost of serving a client. This figure illustrates the annual program costs of supplying a beneficial service to a client. This number attempts to get at the cost of having a monetary impact on a clients' business, although it does not capture the depth of the impact. This is a simplified cost-benefit analysis. It attempts to answer the question: in the most recent year evaluated, for every 1$ spent on program development and operations, how much impact, in terms of profits to assisted client, has the program generated? What was the social return on the program investment for the most recent year the program was evaluated? For example, a ratio of 2, or 200%, would suggest that every program dollar spent resulted in $2 of profits that assisted clients report as being attributable to the use of the service.

10 METHODOLOGY Sustainability Ratios Goal: Achieve supplier sustainability Indicator Proposed Methodologies Interpretation Financial Sustainability = Non-donor Revenues / Total Expenses BDS supplier financial sustainability BDS contribution margin BDS1 Contribution Margin = (SME Revenues from BDS1- BDS1 Direct Expenses)/ Total Expenses BDS2 Contribution Margin = (SME Revenues from BDS2- BDS2 Direct Expenses)/ Total Expenses Financial Sustainability ratio answers the question: last year, what percentage of total expenses of a BDS supplier were covered by non-grant revenue? The contribution margin of each BDS answers the question: Last year, what % of total expenses were covered by this particular BDS? When comparing different BDS, it answers the question, which BDS is more profitable for my institution? And so on. BDS viability BDS1 Viability = Revenues from BDS1/ Direct Expenses for BDS1 BDS1 Viability = Revenues from BDS2/ Direct Expenses for BDS2 This ratio tells us: How profitable was a particular BDS service? Last year, what percentage of the direct expenses to deliver a BDS were covered by the revenues generated for that BDS? And so on. As described above, the PMF is testing three sustainability ratios that provide slightly different perspectives on the same issue: how financially viable is this service without donor subsidies? Each ratio compares costs to revenues, in an attempt to analyze profitability or cost-recovery or sustainability (used interchangeably here). Be aware that in the following instructions, there are different directions for calculating these revenues and expenses for the different ratios. Also, there are different instructions depending on the type of institutional arrangement in the BDS program. Please review the descriptions of different BDS organizations on page seven, and take a moment to decide which type of program you represent, and therefore, which instructions you will follow when there are differences. BDS Supplier Financial Sustainability The BDS supplier Financial Sustainability ratio compares the expense of supplying services, to the non-donor revenue generated from those services. It answers the question: will this institution be able to supply this BDS without receiving direct donor funding? To get the ratio, we first calculate expenses, then revenues, and then the ratio itself. The following ratios should be calculated for the two most recent fiscal years for which complete data is available.

11 1. Calculate Total Expenses. Collect annual expenses for the two most recent years that data is available. For BDS Suppliers: Total Expenses are all of the institutions' expenses, including direct and indirect, operational and developmental, variable and fixed costs, and depreciation. There is no need to break these costs down into any categories. Include in an annex, a copy of annual financial statements. For BDS Suppliers Plus: Total Expenses are all of the BDS program's expenses for the most recent fiscal year, including direct and indirect, operational and developmental, variable and fixed costs, and depreciation. There is no need to break these costs down into any categories. In addition, the Total Expenses include an appropriate proportion of overhead or indirect costs incurred by the institution that manages the BDS program. Report Costs as Follows: Total Program Expenses: Overheads: Total Expenses: Report the rational for the overhead allocation in detail. Include in an annex a financial statement of the program expenses included in Total Expenses. For BDS Facilitators: Work with your BDS supplier partners to calculate the profitability of each BDS supplier you work with. If the BDS supplier exclusively supplies the BDS that your support, then Total Expenses are the supplier's Total Expenses, including direct and indirect, operational and developmental, variable and fixed costs, and depreciation. There is no need to break these costs down into any categories. If the BDS you support is one of many services or line of business, then Total Expenses is the total expense of supplying the BDS, plus an appropriate allocation of overhead or indirect costs incurred by the institution that supplies the BDS. Report Costs as Follows: Total BDS Expenses: Overheads: Total Expenses: Report the rational for the overhead allocation in detail, and if possible a financial statement from the BDS suppliers indicating the line items included in total expenses. For BDS Facilitators/Suppliers: You will report sustainability for both you and your partners, using the above instructions that correspond to the type of program you have and the type of supplier you support. 2. Calculate Non-Donor Revenues Calculate revenues for the two most recent fiscal years. Non-Donor Revenues in this context refers to revenues from private sector actors because the purpose of this ratio is to assess the viability of the BDS without donor funding. In all cases, Non-Donor Revenues should primarily consist of revenues from SMEs. In addition, some revenue may also include: o Revenue from larger enterprises (not charitable donations, but fees for services) For example, a radio program targeting SEs may have large business advertisers.

12 o Revenue from other BDS suppliers (for example if they one BDS supplier pays another to train their staff) o Revenue received from donors through the redemption of vouchers where the vouchers allow entrepreneurs to select their service supplier. Specifically excluded from Non-Donor Revenues in this context, is any donor funding, grants or contracts to deliver services directly to entrepreneurs or to develop BDS or the BDS market. Any funding received from public or private charitable donors should be excluded. The only exception is funding from vouchers. If a donor contracts a supplier to serve SEs or to develop training materials to serve another supplier, this revenue does not constitute Non-Donor Revenues. Note the method you use calculated revenues and what revenues you included and excluded. BDS Suppliers: Report all Non-Donor Revenue as defined above. Suppliers Plus: Report all non-donor BDS program revenue, as defined above. BDS Facilitators: Report non-donor revenues for each of their suppliers separately. If the BDS supplier exclusively supplies the BDS that you support, then Non-Donor Revenues are the supplier's Non-Donor Revenues, as described above. If you provide grants to these suppliers, the grant income should be excluded. If the BDS you support is one of many services or lines of business, then report Non-Donor Revenues for the BDS you support only. BDS Facilitator/Suppliers: Report Non-Donor Revenues, as defined above, for your institution and each of your suppliers, using the relevant corresponding instructions for BDS Suppliers, Suppliers Plus and Facilitators. 3. Calculate the ratio Calculate ratios for the two most recent fiscal years. The profitability ratio is: Supplier profitability = Non-donor Revenues / Total Expenses BDS Suppliers: calculate the ratio for your institution. Suppliers Plus: calculate the ratio for your program. BDS Facilitators: Calculate a separate ratio for each of your partners. BDS Facilitator/Suppliers: Calculate a sustainability ratio for your institution and each of your partners. SEE attached spreadsheet (PMF Sust. & CE Tool) for assistance in calculating and reporting these ratios. 4. Interpret

13 Analyze the financial sustainability ratios. What do they say about your organization and/or program? Is this an appropriate level of sustainability for your institution at this point in time? Why or why not? Were there changes over the two year period? What does this reflect? Are there differences among partners? What does this reflect? 5. Assess the process: Is this a valid indicator for financial sustainability? Was it easy to collect and calculate? Is it useful? BDS Contribution Margin The next two indicators look at the financial viability or profitability of a specific BDS. The first, BDS contribution margin, asks: what proportion of overheads or indirect costs are being covered by profits from this BDS? The formula is: BDS1 Contribution Margin = (Non-Donor Revenues from BDS1-BDS1 Direct Expenses)/ Total Expenses The ratio should be calculated for each service offered in the BDS program. It is most useful for comparing the profitability of specific services within an institution, to help managers decide which services to focus on in order to maximize profits or increase efficiency. 1. Calculate Non-Donor Revenues from BDS: Using the definition of Non-Donor Revenues described above - primarily SME revenues in most cases - identify the revenues paid for each services and report them separately as Revenues BDS1, Revenues BDS 2, etc. BDS Suppliers and Suppliers Plus: Break down revenues by different types of BDS. BDS Facilitators: Work with your partners to break down revenues by different types of BDS. The best strategy is to make a different table for each supplier (not each service), and report each supplier's information separately. SEE annex. 2. Calculate Direct Expenses The definition of direct and indirect expenses varies. "Direct expense" is typically a nonprofit term. The business equivalent is cost of goods sold. It normally includes such expenses as materials, staff, travel and facilities rental, and marketing costs if these are easily divisible for particular services. "Indirect expenses" would then include management time, administration, accounting, monitoring, depreciation, communications, and other costs that are more difficult to break down by particular service. For the purposes of the PMF Field research, each organization may use the method they currently use to divide direct and indirect costs. Please provide a spreadsheet with each line item of Direct Costs specified. Also, were relevant, note how the cost allocation among different BDS was determined. Then, we will compare methods and see whether a common definition is required.

14 BDS Suppliers and Suppliers Plus: Calculate the direct cost for each BDS being analyzed in this research. BDS Facilitators: Calculate the direct cost for each service for each partner. BDS Facilitator/Suppliers: Calculate the direct cost of each service supplied by you and your partners. 3. Calculate the BDS Contribution Margin Using the Total Expense figure from the Financial Sustainability ratio, and the two above figures for direct expenses and BDS non-donor revenue, calculate the following ratio: BDS1 Contribution Margin = (Non-Donor Revenues from BDS1-BDS1 Direct Expenses)/Total Expenses Calculate this ratio for each BDS. For organizations working with suppliers, calculate a set of ratios for each supplier. 4. Interpret Analyze the profitability ratios. What do they say about your organization and/or program and the different services provided? Is this an appropriate level of sustainability for the services at this point in time? Why or why not? Were there major difference among services? Among suppliers? What does this reflect? 5. Assess the process Is this a valid indicator for financial sustainability? Was it easy to collect and calculate? Is it useful? BDS Viability This final sustainability ratio looks at the financial viability or profitability of a specific BDS, regardless of overheads or indirect costs. It asks: what portion of the service's direct costs are being covered by non-donor revenues from this BDS? The formula is: BDS1 Viability = Non-Donor Revenues from BDS1/BDS1 Direct Expenses It should be calculated for each services offered in the BDS program. It is useful for comparing the basic viability of specific services, regardless of the overall institutional costs of providing the service. 1. Calculate the ratio:

15 Using the revenues and direct expenses calculated for particular services (see Contribution Margin above), simply calculate the ratio. Organizations follow the same pattern in report as the one for contribution margin. 2. Interpret Analyze the service viability ratios. What do they say about your organization and/or program and the different services provided? Is this an appropriate level of sustainability for the services at this point in time? Why or why not? Were there major difference among services? Among suppliers? What does this reflect? How do the three ratios differ? When looking at them all together, and consider the program trends, what conclusions can you draw? 3. Assess the process: Is this a valid indicator for financial sustainability? Was it easy to collect and calculate? Is it useful? SEE attached PMF SUST. & CE Tool for assistance in calculating and reporting these ratios.

16 Methodology: Cost-Effectiveness Ratios As described above, the PMF is testing five cost-effectiveness ratios that provide slightly different perspectives on the same issue: What benefits have resulted per $1 (or other currency) invested in the most recent year for which data is available? Each ratio compares costs to results or benefits, in an attempt to analyze cost-effectiveness as a proxy for cost-benefit. Each ratio uses the same calculation of total expenses, which focuses on donor funded expenses. Also, there are different instructions depending on the type of institutional arrangement in the BDS program. Please review the descriptions of different BDS organizations on page seven, and take a moment to decide which type of program you represent, and therefore, which instructions you will follow when there are differences. Finally, it is the case that some participants in the PMF collected significant relevant impact data from their baseline survey that can be used in the calculation of these indicators. If that is the case, please use that data to create the ratios that you can. Otherwise, simply report costs, and comment on the potential viability of the ratios. These ratios will be calculated again following the repeat impact survey in June. SEE the attached spreadsheet PMF Sust & CE Tool - Sheet 2! for helpful spreadsheets and sample tables. Cost per Client: This is annual program expenses divided by the number of businesses acquiring the services that year. 1. Calculate Annual Program Expenses Conduct an impact assessment and determine the fiscal year that most closely corresponds to the time period the for impact assessment. For example, if the impact assessment took place with a baseline of April 2000 and a repeat survey in June, 2001 (as will be the case with most PMF Field testers), then any of the following fiscal years would be appropriate: Sept 1999-Aug 2000; January 1999-January 2000; April April 2001 (If data is available). The following would not be appropriate: April April 1999; September 2000-September If the results of your impact survey correspond to the clients' business results from , then you could use program expense data from fiscal year. Once you have established an appropriate fiscal year, calculate all program expenses. The strategy differs according to different types of programs: BDS Suppliers and Suppliers Plus: Annual Program Expenses are the same as the Total Expenses figure used to calculate sustainability ratios. BDS Facilitators: Annual Program Expenses refers to the facilitator's annual expenses, plus any grants the facilitator has made to partner BDS suppliers. In other words: you are trying to capture the Facilitator's investment in the program, not the investment made by the BDS Suppliers. BDS Facilitator/Supplier: The Annual Program Expenses are your total program expenses for facilitation and supply activities, plus any grants to partner BDS suppliers.

17 GOAL: Improve program cost-effectiveness Indicator Proposed Methodology Interpretation 1)Conduct an impact survey and identify the fiscal year most closely corresponding to the period of the impact survey. Calculate the annual program expenses for that fiscal year, for the BDS initiative for which you are tracking impact. 2)Calculate the ratio: Annual program expenses/ number of customers served that year Annual program expenses per customer served* Annual program expenses per supplier assisted* Annual Program Expenses per firm that "used" a BDS Annual Program Expenses per firm that reported "benefiting from" a BDS. Simplified costbenefit assessment comparing annual program costs to aggregate annual program benefits for entrepreneurs* Annual program costs/ number of suppliers served that year (if you are the only direct provider in your program, the number of suppliers is 1) 1) Estimate the number of firms that used a service that year by multiplying the percent of firms reporting having "used" a service, by the total number of firms assisted that year. 2) Then, divide the Annual Program Expenses by the estimated number of firms that used the service. (Expenses/#firms) 1) Estimate the number of firms that benefited from a service that year by multiplying the percent of firms reporting having "benefited from" a service, by the total number of firms assisted that year. 2) Then, divide the Annual Program Expenses by the estimated number of firms that used the service. (Expenses/#firms) 1)Estimate the total profits for firms assisted that year that may be attributable to the program. To do this, multiply the number of firms served that year by the % in the impact survey who attribute profit change to the service. Then multiply the average annual increase in per firm profits 5 by the estimated number of firms that would attribute change to the service, and add the results. 2)Calculate the ratio of aggregate annual benefits to annual program expenses. Benefits/cost. Express it in a ration and in percentage terms. This figure is the total program cost of delivering a service to one client. How much does it cost to serve one business? This figure is easy to calculate, but used alone would favor programs that serve many, but may not have a significant impact on the firms. This indicator illustrates the cost of assisting each supplier. It should be looked at in the context of the program's effectiveness in developing the BDS Market. This is an attempt to look at the relative cost of that market development strategy. This figure tells us the annual program cost for each client who applied the service in their business. It illustrates the cost of providing a useful service to a client. This number would normally be higher than the cost of serving a client. This figure illustrates the annual program costs of supplying a beneficial service to a client. This number attempts to get at the cost of having a monetary impact on a clients' business, although it does not capture the depth of the impact. This is a simplified cost-benefit analysis. It attempts to answer the question: in the most recent year evaluated, for every 1$ spent on program development and operations, how much impact, in terms of profits to assisted client, has the program generated? What was the social return on the program investment for the most recent year the program was evaluated? For example, a ratio of 2, or 200%, would suggest that every program dollar spent resulted in $2 of profits that assisted clients report as being attributable to the use of the service.

18 2. Calculate Annual Number of Clients: Once you have calculated Annual Program Expenses, calculate the annual number of clients served for the year that most closely corresponds to the fiscal year of the financial data. This is the number of clients acquiring the service from you and your partner BDS suppliers, in the case of facilitators. Although different numbers of clients accessed different services, report the total number of clients acquiring a service. Do not duplicate, however: if a client received 2 services, he should only be counted once. 3. Calculate the Cost per Client Annual Program Expenses / Annual Number of Clients 4. Interpret: What does this figure tell you about the program? 5. Assess: Was this easy to calculate? Is it accurate and fair? Is it a useful indicator for cost-effectiveness? Cost per Supplier This indicator is a basic cost-effectiveness ratio for assessing the impact on the market. 1. Calculate the number of suppliers assisted for the year most closely corresponding to the fiscal year for which financial data is available. 2. Calculate cost per supplier: Annual Program Expenses / Annual Number of Suppliers 3. Interpret: What does this figure tell you about the program? 4. Assess: Was this easy to calculate? Is it accurate and fair? Is it a useful indicator for cost-effectiveness? Cost per Business "Using" and "Benefiting" from the service: In the impact survey, the PMF asks businesses to report whether they are using a service as intended. For example, if they attended a trade show, did they make new contacts for their business? Then it asks whether the business benefited from using that service. For example, did the business make new sales or find a lower cost more reliable supplier as a result of these new contacts? The next two ratios simply put a cost figure to these results: what did it cost to help one business "use" a service or "benefit from" a service? The results will generally be higher than the cost per client figure. The different in the cost per client, cost per firm using and cost per firm benefiting will reflect the extent to which a program is effectively helping clients translate buying a service into applying it to their business and then creating financial benefits from the service.

19 1. Estimate the number of businesses "using" the service. Using the results from the impact survey and the calculated number of businesses acquiring the service, calculate the number of businesses using the service. Identify the percentage of businesses that "used" a service, according to the impact survey. If you have different results for different services, take an average of the different services. Then estimate: Annual Number of Clients "Using" a service = % using * annual number of clients served 2. Calculate the cost per client using the service: Annual Program Expenses / Annual Number of Clients Using the Service 3. Estimate the number of businesses "benefiting from" the service. Using the results from the impact survey and the calculated number of businesses acquiring, calculate the number of businesses benefiting from the service. Identify the percentage of businesses that "benefited from" a service, according to the impact survey. If you have different results for different services, take an average of the different services. Then estimate: Annual Number of Clients "Benefiting" = % benefiting * annual number of clients served 4. Calculate the cost per client benefiting from the service: Annual Program Expenses / Annual Number of Clients Benefiting from the Service 5. Interpret: What do these figured tell you about the program? Do they add anything that the cost/client and impact surveys didn't already tell us? 6. Assess: Was this easy to calculate? Is it accurate and fair? Is it a useful indicator for cost-effectiveness? Simplified, Annual Cost-Benefit Ratio It cannot be overemphasized that this ratio is by no means a thorough cost-benefit analysis, which looks at cumulative, total costs and benefits over time. Rather, it is an annual cost-benefit ratio that takes into consideration only limited benefits that BDS program managers can fairly easily assess on an annual or semi-annual basis. The PMF is attempting to assess the increase in business profits that businesses attribute to the application of the BDS being assessed. It is yet to be determined whether the methodology being used in the impact survey for this data is valid. The following description assumes that the data will be valid. The issue will be revisited after the second impact survey. 1. Estimate Aggregate Program benefits Using data from the impact survey, estimate total program benefits. The impact survey reported average change in business profits, and a percent of businesses that attribute

20 change to the BDS. Use this data as follows to estimate aggregate annual program benefits: Aggregate Annual Program Benefits = Number of clients served * %that attribute change to the service * Average profit increase per firm 2. Calculate simplified cost-benefit ratio: Aggregate Annual Program Benefits / Annual Program Expenses Express the result as a number and a percentage: (If the result is 2, for example) A program investment of $1 (or other currency) resulted in $2 of benefits for businesses, a 200% return in benefits to the business. 3. Interpret: What does this figure tell you about the program? How do the other indicators help to interpret or understand the result? Consider the sustainability and costeffectiveness ratios altogether - what do they tell you about the program as a whole? 4. Assess: Was this easy to calculate? Is it accurate and fair? Is it a useful indicator for cost-effectiveness? SEE the attached spreadsheet PMF Sust & CE Tool - Sheet 2! for helpful spreadsheets and sample tables.

21 DATA PRESENTATION, ANALYSIS & REPORTING As with the other PMF reports, the report on sustainability and cost-effectiveness should report data and analyze findings, but also should comment on the validity, practicality and usefulness of the approach to assessing sustainability and cost-effectiveness. Please include recommendations for improving the process. The following is a suggested outline for the PMF Field Research Sustainability and Cost-Effectiveness Report. 1) Executive Summary. a)pmf Table: (See following page) b) Data Analysis: What does the above data tell the reader about your organizational and program performance? What has been the sustainability trend in the last 2 years? How does the sustainability of different services and suppliers compare? What do the different cost-effectiveness indicators say about program performance? What is the overall cost-effectiveness of the program? c) Methodology Analysis: Overall, are the indicators valid, practical and useful? What were the major strengths and weaknesses of the methodologies used? What major recommendations would you have for changing the methodologies?

22 Indicator BDS Supplier Financial Sustainability: Non-Donor Revenue /Total Program Expenses BDS Contribution Margin: (Non-Donor Revenues from BDS- BDS Direct Expenses)/ Total Expenses Service 1: (insert the name of the service here) BDS Contribution Margin: Service 2: (insert the name of the service here) BDS Contribution Margin: Service 3: (insert the name of the service here) BDS Viability: Non-Donor Revenues from BDS/ Direct Expenses for BDS Service 1: (insert the name of the service here) BDS1 Viability: Service 2: (insert the name of the service here) BDS1 Viability: Service 3: (insert the name of the service here) Financial Sustainability: (insert program name) First Program year (replace with the year) Supplier 1: (insert the name of the supplier here) Supplier 2: (insert the name of the supplier here) 2000 (or replace with 1999 if that is the most recent year) Supplier 3: (insert the name of the supplier here) Cost-Effectiveness: (insert program name), YEAR: Annual program expenses per customer served Annual program expenses per supplier served Annual program expenses per firm that "used" a BDS Annual program expenses per firm that reported "benefiting from" a BDS. Simplified cost-benefit assessment comparing Annual program expenses to aggregate annual program benefits for entrepreneurs - ratio Simplified cost-benefit assessment comparing Annual program expenses to aggregate annual program benefits for entrepreneurs - percent social return on investment. Supplier 4: (insert the name of the supplier here)

23 2) Sustainability Ratios a) Data Presentation & Methodology Description : repeat the above table. Repeat the Sustainability Table above. Follow this table with an explanation of particular expense and revenue allocation procedures used. Include a list of direct costs and indirect costs, and any necessary explanation for putting them into one or the other category. Include the relevant expense reports or spreadsheets in an annex. b) Data Analysis: What does the above data tell the reader about your organizational and program performance? What has been the sustainability trend since the over the last 2 years? How does the sustainability of different services compare? How does the sustainability of different suppliers compare? Share any other interesting observations about your program based on this data. c) Methodology Analysis: Overall, are the indicators valid, practical and useful? What were the major strengths and weaknesses of the methodologies used? Comment on expense allocation, revenue allocation and ratio calculation. What major recommendations would you have for changing the methodologies? Which sustainability ratios are more valid, practical and useful, and why? 3) Cost-Effectiveness Ratios a) Data Presentation & Methodology Description Repeat the Cost-Effectiveness table from the Executive Summary. Follow this table with a description of how total program expenses were calculated. Also, describe how the numbers of clients and suppliers serviced, and the number that "used" and "benefited from" the service were estimated. And, present how you estimated the aggregate annual profit increase that businesses attributed to the service. Skip this if you did not gather this data in the baseline impact survey. Include any additional methodological description if your methodology varied from that described in the guidance. b. Data Analysis: What does the above data tell the reader about your organizational and program performance? What do these cost-effectiveness ratios illustrate about your program? How does the cost per client services compare with the cost per client that used and benefits form the service? What does this say about your program? Share any other interesting observations about your program based on this data. c. Methodology Analysis: Overall, are the indicators valid, practical and useful? What were the major strengths and weaknesses of the methodologies used? Comment on expense allocation, benefit estimating, and ratio calculation. Given the validity of your impact data, how valid are the cost-effectiveness ratios? Do you have other data that supports or refutes the above findings? What major recommendations would you have for changing the methodologies? 4) Summary of recommendations: Summarize the aspects of the indicators and methodologies that were valid, practical and useful, and make recommendations for either eliminating particular indicators or adjusting methodologies and indicators to make them more valid, practical and useful.