Guide to Core Output and Outcome Indicators for IFC Technical Assistance Programs

Transcription

1 Guide to Core Output and Outcome Indicators for IFC Technical Assistance s Revised January 26, 2006

2 Table of contents I. Introduction...3 Purpose...3 Basic concepts...3 Outline...5 II. Core Indicators...6 Output Indicators for IFC technical assistance projects...7 Outcome and Impact Indicators for Access to Finance...9 Outcome and Impact Indicators for Business Linkages...13 III. Outcome Indicators for Business Enabling Environment...17 Data collection...21 Overview Surveys...21 Secondary sources...22 Budget guidelines...23 Appendix A. Logic Models...24 Appendix B. Definitions of Key Terms...28 Appendix C. A Note on Impact Assessment

3 I. Introduction Purpose 1. This guide presents indicators that can be used to assess technical assistance programs that are geared to promoting private sector development, including but not limited to programs directed toward small and medium. 1 Specifically, the guide provides a core set of output, outcome and impact indicators for three types of technical assistance programs access to finance, business linkages, and business enabling environment along with explanatory notes and guidance on data collection procedures. 2. The purpose of standard core indicators is three-fold: Provide an objective basis for assessing performance of IFC projects; Provide a solid foundation for management decisions ; and Facilitate learning within IFC and external stakeholders. 3. The core indicators are based on discussions with program managers and M&E staff, a review of program documents, and a review of literature on similar programs undertaken by other organizations. In defining the indicators, emphasis has been placed on trying to ensure that indicators are relevant to the types of technical assistance projects most frequently undertaken by IFC in developing countries, are valid with respect to reflecting the underlying concept that is meant to be measured, are reliable in terms of minimizing potential measurement error, and are practical for facilities and regional offices to use given the effort required to collect necessary data. Basic concepts Indicators are derived from program logic models. The indicators are based on program logic models developed for each of the three major programs. 2 These models describe the sequences of cause and effect relationships that link IFC program activities to intended impacts. Each model has five basic components as illustrated in Figure 1. Inputs refer to the resources used in program activities. Activities are the actions taken or work performed in particular projects using specified inputs. IFC technical assistance projects include activities such as assessments, advisory services, training, and public awareness campaigns. These activities are intended to result in outputs such as reports, advice, training events, and media coverage. In turn, these outputs are expected to yield certain outcomes in terms of changes in knowledge, behavior and performance among beneficiaries in the target population. Finally, it is anticipated that programs will generate development impacts including higher productivity, greater income, and economic growth. The indicators presented in the guide focus on outputs of IFC program activities as well as associated outcomes and impacts. 1 Initially the effort to define core indicators focused on programs directed to SMEs, but was subsequently broadened to include programs to promote the development of private regardless of size. 2 Appendix A contains detailed program logic models for each of the three program areas. 3

4 Figure 1. Basic Logic Model If If If If Target Population Inputs Then Activities Then Outputs Then Outcomes Then Impacts ASSUMPTIONS The indicators account for the fact that IFC may provide services directly to a specific set of beneficiaries or indirectly through intermediaries. In many cases, IFC staff or third parties under contract to IFC provide technical assistance directly to private. However, it is important to note that IFC also frequently works with intermediaries such as financial institutions, government agencies, large multinational corporations, and providers of business development services. IFC staff and/or third parties under contract to IFC undertake program activities that are directed at these intermediaries with the expectation that these intermediaries will in turn carry out activities that will benefit private, including SMEs. As such, changes in the magnitude and nature of activities undertaken by intermediaries are considered as outcomes of IFC technical assistance projects. Whether a particular indicator is defined as an output or outcome depends on how the project is carried out. Some indicators may not be relevant to a specific facility given the facility s particular project mix. managers are meant to select indicators included in this guide that are relevant to their programs and for which data are available at a reasonable cost. Not all indictors need to be adopted by every facility. For example, if a facility does not carry out projects that aim to reduce the number of days required to register a business it would not need to track this indicator. However, facilities that do carry out such projects should use the indicators included in the guide that deal with improvements in the business registration process to ensure that measures are consistent across IFC. In decided who to measure, careful consideration needs to be given to the definition of the target population and program participants. Outcome and impact indicators are used to measure different characteristics of entities within specified populations. The target population represents the intended beneficiaries of the program. These could be individuals, financial institutions, BDS providers, business organizations and/or private (including SMEs). As noted above, these entities can be either direct or indirect beneficiaries of the intervention. s may be directed toward one or more target populations. participants are members of the target population that receive services or are otherwise affected by the program. Some interventions are full-coverage programs in that every member of the target population is involved. This is most likely in programs geared to reforming laws and regulations. In full coverage program, it is important to assess the affects of the intervention on all 4

5 entities within the relevant population. However, most technical assistance projects involve a sub-set of the target population that voluntarily agrees to participate in the program. In partial coverage programs, it is important to assess the affect of the intervention on the sub-set of the target population that actually participated in the program. 3 Indicators by themselves do not address the issue of attribution. For the most part, the outcome and impact indicators presented in the guide represent gross program effects, i.e., they do not provide an estimate of the net result of IFC interventions. As such they are only intended for monitoring purposes to provide a sense of whether members of the target population have been reached and have improved performance on various dimensions. Attributing observed changes to the IFC intervention would require more rigorous studies that involve comparison with a counterfactual the hypothetical situation that would have occurred in the absence of the program. Good impact assessments use random assignment, constructed controls, and/or other approaches to isolate the impact of the program from other extraneous factors affecting outcomes. 4 It is envisioned that IFC will undertake more comprehensive evaluations of programs, including more rigorous impact assessment. Multiple measures are needed to calculate changes over time. The outcome and impact indicators included in the guide are expressed in terms of a level as of a specific date. The intention is for programs to establish a baseline and assess changes in the level of particular indicators over time. Changes can be calculated on an absolute or percentage basis. Outline 4. The guide is divided into three sections. Following this introduction, Section II presents core indicators for each of the three programs. Section III provides guidance on data collection, outlining requirements for program, s, and secondary data. 5. Three appendices are attached. Appendix A contains the detailed program logic models. Appendix B presents definition of key terms. Finally, a note on impact assessments is presented in Appendix C. 3 Partial coverage programs provide an opportunity to compare participants to similar, non-participants as a means of assessing net impacts. 4 A brief note on methods for assessing impacts is presented in Appendix C. 5

6 II. Core Indicators 6. This section contains core output, outcome and impact indicators for IFC Technical Assistance Projects. The section is divided into four parts: Output indicators for technical assistance projects Outcome and impact indicators for access to finance Outcome and impact indicators for business linkages Outcome and impact indicators for business enabling environment 6

7 Output Indicators for IFC technical assistance projects 7. Output indicators aim to measure the magnitude of the outputs produced directly by IFC and/or third-parties under contract to IFC in technical assistance projects. These indicators reflect outputs of activities carried out during a given quarter. Definitions of key terms are included in Appendix B. Activity Indicator Data Source Assessments Number of reports produced by IFC and third-parties under contract to IFC in the quarter Advisory services Number of advisory service hours provided by IFC and third-parties under contract to IFC in the quarter (by type of beneficiary) Number of unique organizations to which advisory services were provided by IFC and third-parties under contract to IFC in the quarter (by type of beneficiary) Training Public awareness campaign Number of unique organizations to which advisory services were provided by IFC and third-parties under contract to IFC in the quarter that had not received services previously, i.e., a new client (by type of beneficiary) Number of unique organizations to which advisory services were provided by IFC and third-parties under contract to IFC in the quarter that had received services previously, i.e., a repeat client (by type of beneficiary) Number of new laws/regulations drafted or contributed to the drafting. Number of laws/regulations to which improvements have been recommended. Number of training events (courses, seminars, workshops, ) conducted by IFC and third-parties under contract to IFC in the quarter Number of participants in training events conducted by IFC and third-parties under contract to IFC in the quarter (by type of beneficiary) Number of participant-training hours provided by IFC and third-parties under contract to IFC in the quarter (by type of beneficiary) Number of unique organizations that sent individuals for training conducted by IFC and third-parties under contract to IFC in the quarter (by type of beneficiary) Number of unique organizations that sent individuals for training conducted by IFC and third-parties under contract to IFC in the quarter that had not sent individuals previously, i.e., a new client (by type of beneficiary) Number of unique organizations that sent individuals for training conducted by IFC and third-parties under contract to IFC in the quarter that had sent individuals previously, i.e., a repeat client (by type of beneficiary) Number of substantive media reports produced in association with IFC and third-parties under contract to IFC in the quarter Number of press releases distributed Number of TV/radio spots aired Number of unique visitors to Web site developed by IFC and third-parties under contract to IFC in the quarter (if website is a result of the project being monitored) Number of public information events held or sponsored by IFC and thirdparties under contract to IFC in the quarter Number of participant at public information events held or sponsored by IFC and third-parties under contract to IFC in the quarter 7

8 8. The output indicators for advisory services and training ask for a breakdown by type of beneficiary, i.e., BDS providers, public institutions, business organizations, financial institutions, and. With respect to the latter, if required, facilities should also keep track of outputs by size of firm to determine the extent to which services have been delivered to SMEs. 9. While advisory services and training are treated separately, a particular organization may receive both types of services during the quarter. As such, adding the number of organizations that receive advisory services to the number of organizations that sent people for training would overstate the number of organizations that actually received services during the period some would be double counted. 10. The indicator number of training participants refers to the number of people that attended training events in the quarter. Although, it would be preferable to count the number of unique individuals that participated in training events this would require an information system that assigns a unique code to individuals and an ability to determine whether the same individual attended more than one training event in the quarter. Client Satisfaction with Training or Advisory Services 11. Facilities are interested in measuring the level of satisfaction of organizations that received services. While a five-point scale should be used in the, the indicator sums the percentage very satisfied and satisfied. Construct Indicator Data Source Satisfaction with services provided by IFC and third-parties under contract to IFC Percent of clients indicating being very satisfied or satisfied with the service (Five-point scale where 5=very satisfied, 4=satisfied, 3=neutral, 2=dissatisfied, and 1=very dissatisfied) Client satisfaction 8

9 Outcome and Impact Indicators for Access to Finance 12. Core outcome and impact indicators for the access to finance program aim to measure characteristics of financial markets and financial institutions. With respect to the latter, some projects may be directed at all financial institutions operating in the market; others may only involve a sub-set of institutions that receive services directly from IFC and/or third parties under contract to IFC. IFC facilities need to consider the purpose of particular projects in determining the population to be measured. 13. Although some facilities expressed the need to assess access to financing for private regardless of size, impact indicators should be broken down by size of loan as proposed by the Financial Infrastructure & Institution Building Department of IFC Measures of profitability of financial institutions such as the return on assets were excluded from the list of core indicators due to perceived difficulty in obtaining necessary information. 5 Access to Finance: Measuring Results to Improve Technical Assistance, Nataliya Mylenko, May

10 Outcome and Impact Indicators for Access to Finance Construct Indicator Data Source Population Outcomes Change in laws and regulations related to FI Learning in terms of knowledge and skills Introduction of new financial products Competition in financial markets Establishment of collateral registries or credit bureaus Use of collateral or credit information Number of laws and regulations promulgated (new or amended) during the year where IFC or third-parties under contract to IFC played a role in the process Percent of participants strongly agreeing or agreeing with the statement that they have obtained new knowledge and skills as a result of training (Fivepoint scale where 5=strongly agree, 4=agree, 3=neutral, 2=disagree, and 1=strongly disagree) Number of participants reporting having obtained new knowledge and skills as a result of training New financial products (standard checklist) introduced into the market during the year Standard checklist: Commercial loans Leasing Factoring Microfinance products Housing finance (mortgage and home equity) Insurance Other. Please specify Number of financial institutions providing specific financial products (standard list) as of year end Standard list: Commercial loans Leasing Factoring Microfinance products Housing finance (mortgage and home equity) Insurance Other. Please specify Number of active participants in relevant financial markets, e.g., commercial lending, leasing, equity, etc. Number of collateral registries or credit bureaus legally registered as of year end Number of FIs that purchased credit reports during the year Training Participant Survey Training Participant Survey FI Regulatory authority or FI Regulatory authority or FI Regulatory authority Credit bureau or FI N/A Training participants Training participants Financial institutions / financial institutions Financial institutions Financial institutions N/A Financial institutions 10

11 Number of new loans that were secured by registered capital during the year (by financial product) Collateral registry or FI Financial institutions Process improvement in participating financial institutions Number of participating FIs reporting change in business processes (standard list) during the year Standard list: Adopted new management practices Adopted good corporate governance practices Adopted new loan approval process Adopted new technologies Introduced new or improved products Introduced new distribution channels Entered new markets Other. Please specify Proportion of participating FIs reporting change in business processes (standard list) during the year FI FI financial institutions financial institutions Standard list: Adopted new management practices Adopted good corporate governance practices Adopted new loan approval process Adopted new technologies Introduced new or improved products Introduced new distribution channels Entered new markets Other. Please specify Number of participating FIs reporting improved performance (standard list) during the year FI financial institutions Standard list: Higher quality of portfolio Lower transaction costs Faster turnaround on loan processing Other. Please specify Proportion of participating FIs reporting improved performance (standard list) during the year FI financial institutions Standard list: Higher quality of portfolio Lower transaction costs 11

12 Rate of non-performing loans Faster turnaround on loan processing Other. Please specify Value (US$) of non-current loans / gross loan portfolio by financial product as of year end FI Financial institutions / financial institutions Impacts Supply of credit to Value (US$) of outstanding loans by size of loan as of year end (by financial product) Number of outstanding loans by size of loan as of year end (by financial product) Value (US$) of loans disbursed by size of loan during the year (by financial product) Number of loans disbursed by size of loan during the year (by financial product) FI FI FI FI Financial institutions / financial institutions Financial institutions / financial institutions Financial institutions / financial institutions Financial institutions / financial institutions 12

13 Outcome and Impact Indicators for Business Linkages 15. Outcome and impact indicators for business linkages should be applied to community/cluster development and supply chain development projects (as detailed in the program logic model). All projects undertaken under the business linkages program are partial coverage with certain organizations in the target population electing to participate. Indicators focus on measuring results among participants. 16. While not included in the list of core indicators, some facilities may also want to assess changes in sales, employment and profitability among the large firms at the center of supply chain projects. 13

14 Outcome and Impact Indicators for Business Linkages Construct Indicator Data Source Population Outcomes Advisory services and/or training delivered by BDS providers to Learning in terms of knowledge and skills Financing provided to participating Business process improvement in participating Number of unique assisted and trained by participating BDS providers during the year Percent of participants strongly agreeing or agreeing with the statement that they have obtained new knowledge and skills as a result of training (Fivepoint scale where 5=strongly agree, 4=agree, 3=neutral, 2=disagree, and 1=strongly disagree) Number of participants reporting having obtained new knowledge and skills as a result of training Number of unique participating that received external financing facilitated through the program during the year Proportion of unique participating that received external financing facilitated through the program during the year Value (US$) of loans disbursed by financial institutions or large firm to participating during the year Value of outstanding SME loan portfolio for SMEs that received financing through the program Number of unique participating reporting change in business processes (standard list) during the year Standard list: Adopted new management practices Adopted new process technologies Adopted measures to improve working conditions Adopted measures to reduce environmental impacts Introduced new or improved products Introduced new distribution channels Entered new markets Other. Please specify Proportion of unique participating reporting change in business processes (standard list) during the year Standard list: Adopted new management practices Adopted new process technologies Adopted measures to improve working conditions Adopted measures to reduce environmental impacts Introduced new or improved products BDS Training participant Training participant BDS providers Training participants Training participants N/A N/A N/A N/A 14

15 Introduced new distribution channels Entered new markets Other. Please specify Number of unique participating reporting improved performance (standard list) during the year Standard list: Higher quality Faster order-to-delivery time Higher rate of on-time delivery Lower unit cost Higher productivity Other. Please specify Proportion of unique participating reporting improved performance (standard list) during the year Standard list: Higher quality Faster order-to-delivery time Higher rate of on-time delivery Lower unit cost Higher productivity Other. Please specify Sustainable BDS providers Percentage of BDS total costs covered by fees paid by clients during the year BDS BDS providers Establishment of Supply Chain Development in Large Firm Number of supply chain development programs established during the year N/A 15

16 Impacts -wide performance of participating Purchases by large firm from local SME suppliers Total number of employees of participating as of year end Total gross annual sales (US$) of participating during the year Total value (US$) of gross annual sales of participating to large firm during the year Percent of gross annual sales of participating to large firm during the year Number of participating reporting an increase in gross annual sales compared to previous year Proportion of participating reporting an increase in gross annual sales compared to previous year Number of participating reporting an increase in net profit margin compared to previous year Proportion of participating reporting an increase in net profit margin compared to previous year Number of participating reporting an increase in employment compared to previous year Proportion of participating reporting an increase in employment compared to previous year Number of local suppliers from which large firm purchased goods and services during the year Number of local SME suppliers with active contracts with large firm Purchases (US$) from local suppliers by large firm during the year Large firm Large firm Large firm large firms large firms large firms 16

17 Outcome Indicators for Business Enabling Environment 17. Business Enabling Environment program includes a broad range of projects classified as business registration, business licenses and permits, business inspections, contract enforcement, and property registration. As noted below, indicators need to be tailored to the particular type of project undertaken by IFC. 17

18 Outcome Indicators for Business Enabling Environment Construct Indicator Data Source Population Outcomes Learning in terms of knowledge and skills Changes in the law and regulations related to business enabling environment Changes in relevant administrative procedures Percent of participants strongly agreeing or agreeing with the statement that they have obtained new knowledge and skills as a result of training (Five-point scale where 5=strongly agree, 4=agree, 3=neutral, 2=disagree, and 1=strongly disagree) Number of participants reporting having obtained new knowledge and skills as a result of training Number of laws and regulations promulgated (new or amended) during the year where IFC or third-parties under contract to IFC played a role in the process (by BEE project type) BEE project type: Business registration Business licenses and permits Business inspections Contract enforcement Property registration Other. Please specify Number of administrative procedures changed during the year where IFC or thirdparties under contract to IFC played a role in the process (by BEE project type) Training participant Training participant Training participants Training participants N/A N/A Transaction cost for specific procedures BEE project type: Business registration Business licenses and permits Business inspections Contract enforcement Property registration Other. Please specify Starting a business and dealing with licenses: Official cost of each procedure (as a percentage of income per capita) Registering Property: Official cost (as a percentage of the property value) Enforcing contracts: Official costs (as a percentage of the debt value) Average cost born by companies to complete specific procedures, including government fees, payments to attorneys and other third-parties, and unofficial payments to government officials. (by BEE project type) BEE project type: Business registration Business licenses and permits Doing Business N/A s that completed procedures in the last 12 months 18

19 Business inspections Contract enforcement Property registration Other. Please specify Days required to complete specific procedures Corruption Policy advocacy by participating business organizations Sustainable business organizations Average time, in calendar days, spent to complete each procedure (by BEE project type) BEE project type: Business registration Business licenses and permits Business inspections Contract enforcement Property registration Other. Please specify Percentage of firms that say that corruption is a major or severe obstacle to the operation and growth of their businesses Percentage of firms that made unofficial payments to public officials during the process (by BEE project type) BEE project type: Business registration Business licenses and permits Business inspections Contract enforcement Property registration Other. Please specify Value of unofficial payments made by firms to public officials Number of participating business organizations playing an active role in policy advocacy during the year Proportion of participating business organizations playing an active role in policy advocacy during the year Number of members paying membership fees during the year Doing Business Investment Climate Survey Business organization Business organization Business organization N/A s that completed procedures in the last 12 months N/A s that completed procedures in the last 12 months s that completed procedures in the last 12 months business organizations business organizations business organizations 19

20 Proportion of members paying membership fees during the year Percentage of business organizations total annual costs covered by membership or service fees paid by during the year Business organization Business organization business organizations business organizations Impacts Rate of business registration Number of businesses registered during the year Regulatory authority Private sector productivity and growth Percentage of enterprise reporting that the business enabling environment is conducive to their productivity and growth Survey N/A s that completed procedures in the last 12 months 20

21 III. Data collection Overview 18. Successful measurement depends on the quality of data collected through program, s, and secondary sources. Data should be collected in a consistent manner using agreed definitions and procedures, and stored in appropriate computer databases to facilitate data access, analysis and reporting. 19. The following table lists the sources of data needed to calculate core indicators and the recommended frequency of data collection efforts. detailing the nature and magnitude of activities undertaken by IFC and associated outputs should be continuously updated. Surveys used to assess client satisfaction and learning outcomes should be conducted upon project completion as needed. Other s should be undertaken before programs are initiated to establish needed baselines and repeated annually (as budget allows) in order to monitor changes. Data should also be collected from secondary sources on an annual basis. Data sources Surveys Client satisfaction Training participants BDS providers Business organizations Financial institutions s Secondary sources Collateral registries Credit bureaus Regulatory authorities World Bank - Doing Business World Bank - Investment Climate Frequency of data collection Ongoing Upon project completion Upon completion of training Baseline and annually thereafter Baseline and annually thereafter Baseline and annually thereafter Baseline and annually thereafter Baseline and annually thereafter Baseline and annually thereafter Baseline and annually thereafter Baseline and annually thereafter Baseline and annually thereafter 20. IFC facilities should maintain complete and accurate. This should include data on the characteristics of organizations receiving technical assistance from IFC or third parties under contract to the IFC, including intermediaries and private. It should also include data on particular projects, including the type of activity (assessment, advisory services, training, and information dissemination), participants, service providers, date of initiation and completion, and budget expenditures. Surveys 21. As noted above, IFC will need to conduct a variety of s to collect requisite data. In order to ensure the quality of data, the following procedures should be followed: Questionnaires. IFC facilities should use instruments that contain questions needed to obtain data required for relevant indicators. Questions should be worded in the same 21

22 manner with any translations checked to ensure that meanings have not been altered inadvertently. (This will require the development of standard instruments.) Sampling. The goal is to have a sample that is representative of the population and therefore can be used to make valid generalization. Unless a census is appropriate, IFC facilities should a random sample of organizations drawn from the appropriate set of program participants or the target population as a whole. 6 A random sample is where each entity in the sample frame has a known and independent probability of being selected for the sample. 7 The size of the sample should be large enough to provide sufficient statistical power. 8 Although there are no formal standards for statistical power, IFC facilities should aim to draw a sample that would provide a power of 0.8 or greater. Administration. Given the nature of the information sought, most s should be administered in-person (as opposed to mail or telephone) with a strict promise to protect the confidentiality of the respondent and their responses. Field personnel should be trained to conduct the s. All should seek to achieve a high response rate (at least 60 percent) to reduce potential response bias. To help ensure a high response rate, IFC facilities should obtain the commitment of participants to respond to s as a condition of program participation. Data entry. IFC facilities should establish specific procedures for dealing with completed s. This includes tracking responses so that individuals failing to respond initially can be contacted and encouraged to complete the questionnaire. The quality of data entry should be verified by checking all or a sample of questionnaires for accuracy and by carefully examining data for responses that are not consistent. All questionable entries should be checked for problems and verified. Original copies of written questionnaires should be kept on file. Survey schedule. March is generally a good time to administer s to businesses in that it allows sufficient time for companies to close their books after the end of the fiscal year. (Most fiscal years end December 31 st.) Secondary sources 22. Data required to calculate certain indicators will need to be obtained from secondary sources such as collateral registries, credit bureaus, regulatory authorities, and s conducted by other parts of the World Bank. IFC will need to work with these organizations to ensure that data are accurate and provided in a consistent manner. 6 To facilitate s, agreements should be put in place with intermediaries to provide a complete list of that have received services along with contact information. 7 Other sampling methods such as convenience sampling or purposive sampling are likely to be biased. 8 The power of a statistical test is the probability that the test will reject a false null hypothesis (Type II error). Higher power increases the chance of obtaining a statistically significant result when the null hypothesis is false. Statistical power depends on the degree of variance in the underlying data, the level of confidence desired, and the effect size that researchers want to be able to detect. 22

23 Budget guidelines 23. The budget for data collection and analysis should be commensurate with the scale of the project and/or its importance. As a general rule of thumb, M&E budgets should be in the order of three to five percent of the total program budget. 23

24 Appendix A. Logic Models PLM Access to Finance PLM Business Linkages PLM Business Enabling Environment 24

25 Activities Outcomes Impacts Outside boundaries of accountability 25

26 Activities Outcomes Impacts Outside boundaries of accountability COMMUNITY DEVELOPMENT Advisory services and/or training delivered directly to within targeted geographic/ industrial cluster Development of local infrastructure Lower cost of production for related goods and services Brokerage deals with financial institutions Financing provided to participating Advisory services and/or training delivered directly to BDS providers Establishment of new BDS providers Change in business processes in BDS providers Advisory services and/or training delivered to participating Change in business processes and/or new capital investment by participating Improved performance of participating Higher Quality Faster Turnaround Lower Cost Greater Capacity IFC invesment Establishment / expansion of large firm Sustainable BDS providers Increased demand for intermediate goods and services Increased purchases by large firm from local suppliers and/or Increased sales of large firm through local distributors Improved enterprise-wide performance of participating Productivity Profitability Sales Employment Higher income (value added) in region SUPPLY CHAIN DEVELOPMENT Adoption of targets for local content by large firm Advisory services and/or training delivered directly to large firm Establishment of supplier development program in large firm (inc, certification process and criteria Identification of non-core businesses in large firm Establishment of franchise program or decision to spinoff retail operations (inc, certification process and criteria) Advisory services and/or training delivered to suppliers and/ or distributors Financing provided to suppliers and/or distributors by large firm Change in business processes and/or new capital investment by participating Improved performance of participating Higher Quality Faster Turnaround Lower Cost Greater Capacity Increased in number of qualified suppliers and/ or distributors Lower input / distrbution costs Increased productivity in large firm Increased sales of large firm 26

27 Activities Outcomes Impacts Outside boundaries of accountability LEGAL AND REGULATORY REFORM Advisory services delivered to Executive / Legislative entities Greater awareness of need for reform and knowledge of specific legal and regulatory matters BUSINESS REGISTRATION AND BUSINESS LICENSES / PERMITS Changes in laws and regulations related to business registration and business licenses / permits Advisory services and/or training delivered to registration and business licenses /permits agencies Public awareness campaign Implementation of new administrative procedures Increased knowledge of procedures and rights Lower Fees Lower transaction cost Fewer required days Lower corruption Lower cost of business registration and business licenses / permits Increased rate of business registration (Lower rate of informality) Assessment Studies, e.g., Investment Climate Assessments / Doing Business s / SME Maps / Etc. BUSINESS INSPECTION Changes in laws and regulations related to business inspection Advisory services and/or training delivered to inspection agencies Implementation of new inspection procedures Fewer inspections Lower transaction cost Lower cost of business inspection Public awareness campaign Increased knowledge of procedures and rights Lower corruption CONTRACT ENFORCEMENT Changes in laws and regulations related to contract enforcement Lower court fees Increased productivity and growth of Advisory services and/or training delivered to legal institutions Implementation of new contract enforcement procedures Lower transaction cost Fewer required days Lower cost of contract enforcement BUSINESS ORGANIZATION DEVELOPMENT Advisory services and/or training delivered to business organizations Establishment of new business organizations Change in business processes in business organizations Policy advocacy by business organizations Public awareness campaign PROPERTY REGISTRATION Changes in laws and regulations related to property registration Advisory services and/or training delivered to property agencies Public awareness campaign SECTOR-SPECIFIC Increased knowledge of procedures and rights Implementation of new property registration procedures Increased knowledge of procedures and rights Lower corruption Reliable title Lower fees, transfer taxes, stamp duties Lower transaction cost Fewer required days Lower corruption Lower cost of establishing and transferring title Greater ability to secure financing Increased Investment Sustainable business organizations Changes in laws and regulations related to specific industries Specific firm-level outcomes 27

28 Appendix B. Definitions of Key Terms Term Activities Advisory services Advisory service hour Administrative procedures Assessments Beneficiaries Employees s Gross loan portfolio Impacts Inputs Loans disbursed Loan size New client New products Definition Tasks performed to produce outputs using resources and methods. Consulting services provided to an organization on an individual or group basis. Advisory services are measured in hours. If conversion from days to hours is needed, an 8-hour day should be used. Advisory service time includes time spend in all activities specified under the terms of reference, e.g. report preparation, meetings (in-person or over telephone), and travel. Steps required to comply with a particular regulation, e.g., business registration, licensing and permitting, inspection, or contract enforcement. Reports that examine issues related to particular regions, markets, and/or industries. This includes market studies, needs assessment, and/or policy studies. (Other terms may be used by particular facilities.) Individuals or organizations that are intended to benefit directly or indirectly from the intervention. Types of beneficiaries include: BDS providers, public institutions, business organizations, financial institutions, and. Full- and part-time workers as of a specific date or pay period, e.g., the week of March 12 th s are businesses. To be classified within a specific category, an enterprise must have at least two of the three characteristics (a,b,c): Microenterprise (a) 10 or less Employees (b) US$ 100,000 or less in Total Assets (c) US$ 100,000 or less in Total Annual Sales Small enterprise (a) 10 < Employees 50 (b) US$ < Total Assets US$ 3 million (c) US$ < Total Annual Sales US$ 3 million Medium enterprise (a) 50 < Employees 300 (b) US$ 3 million < Total Assets US$ 15 million (c) US$ 3 million < Total Annual Sales US$ 15 million The outstanding principal balance of all outstanding loans of a particular financial institution. This includes current, delinquent and restructured loans. Final intended outcomes with respect to benefits accruing to the target population and/or broader economy. Financial, human and material resources used to perform activities. Value of all loans paid out during a specified period of time regardless of their current condition (i.e. performing, non-performing or written off). Loans are classified in three categories: micro (< US$ 10,000), small (<US$100,000) and medium (<US$ 1 million for most countries and US$2 million for some more advanced countries, such as Argentina, Brazil, Chile, China, Colombia, India, Korea, Mexico, Morocco, Peru, Russia, South Africa, Thailand, Tunisia, Turkey, and all EU accession countries- Poland, Hungary, Czech Republic, Slovakia, the Baltics and Slovenia). An organization (e.g. enterprise, financial institution, business organization, BDS provider, government agency) that received technical assistance from IFC or third-party providers under contract with IFC in the period, but had not received assistance previously. In the case of Access to Finance projects, the introduction of new products 28

29 introduced to the market Non-performing loans (NPL) Outcomes Outputs Outstanding loans Participant-training hour Participants Policy advocacy Population Project Public information events Region Repeat client to the market denotes the introduction of leasing, factoring, mortgage or other major financial instrument into a particular market for the first time. (Also know as Portfolio at Risk). Loans outstanding that have one or more installments of principal past due by at least 90 days. This includes the entire unpaid principal balance, i.e. past due and future installments but not accrued interest. (Restructured loans may be included in the calculation of NPL since this type of loan carries higher risk than current loans.) Changes in knowledge, behavior (new practices) and performance of direct and indirect beneficiaries. Products of program activities. Loans that have not been fully repaid or written off. It is an hour of training received by an individual during a specific training event. Total participant-training hours for a single event are calculated by multiplying the total number of participants times the number of hours of inclass instruction. Individuals or organizations that have received assistance through the program. Organizations include, government agencies, financial institutions, business organizations, and BDS providers. Activities undertaken to influence the adoption of particular laws and/or regulations. Policy advocacy may include lobbing and public awareness campaigns. The group targeted by the intervention to which results could be generalized. It is the group of individuals or organization from which a sample is drawn. (Technically the sample is drawn from a sample frame.) An intervention that includes a set of planned activities designed to achieve established objectives within a given budget and specified period of time. Activities undertaken as part of public awareness campaigns. IFC may sponsor, organize, and/or participate in these events. The geographic area that is the subject of the IFC intervention. A region could be a country, state, city or other area. An organization (e.g. enterprise, financial institution, business association, BDS provider, government agency) that has received assistance on more than one occasion. SME loans As a proxy, loans to s are defined by the size of the loan: micro (< US$ 10,000), small (<US$100,000) and medium (<US$ 1 million for most countries and US$2 million for some more advanced countries, such as Argentina, Brazil, Chile, China, Colombia, India, Korea, Mexico, Morocco, Peru, Russia, South Africa, Thailand, Tunisia, Turkey, and all EU accession countries- Poland, Hungary, Czech Republic, Slovakia, the Baltics and Slovenia) Substantive media reports Supply chain development programs Surveys Target population Stories or published interviews focusing on projects undertaken by IFC. Substantive media reports can be published through radio, TV, newspapers, or other media. A program organized within a large firm to procure goods and services from local providers in which the firm provides some form of assistance to the suppliers. The intervention would take place at any point of the supply chain to improve quality, costs and delivery issues. Data collection techniques designed to collect standard information from a large number of subjects. Surveys may include polls, mailed questionnaires, telephone interviews, or face-to-face interviews. Intended direct and/or indirect beneficiaries of programs. Target populations include BDS providers, government agencies, financial institutions, business organizations, and private. 29

30 Training events Unique Unique visitors to Web site An activity intended to build the knowledge or enhance the skills of participants. Only training activities with duration of three or more hours should be included. An event is defined by terms of reference for training. The term unique refers to a particular company or organization. Unique implies that the entity should be counted only once. This will require an information system that includes a unique identifier for each company and organization. Represents an unduplicated (counted only once) visitor to a Web site over the course of a specified period, based on hostname or numerical IP address. The implementation of specialized software will be required to track unique visitors. 30

31 Appendix C. A Note on Impact Assessment 24. Impact assessment are undertaken to find out whether a particular intervention actually produced desired results. In this regard, impact assessments are concerned with establishing causality. In demonstrating that a particular intervention resulted in a specific outcome, certain conditions need to be met. 9 First, changes engendered through the intervention have to be shown to produce the effect put another way, the outcome must be responsive to the intervention. Second, plausible alternative explanations for the observed outcome have to be ruled out rival hypotheses must be disproved. Third, the mechanism by which the outcome was produced has to be explained a theory linking the intervention to the outcome must be articulated. Finally, it must be possible to replicate the results in similar settings. With proper research, apparent correlations can be translated into credible causal explanations. 25. Impact assessments center on a comparison of the situation with the intervention to what would have been had there been no intervention at all. The difference in resulting outcomes between these two states constitutes the impact of the intervention as illustrated in the following figure. 10 While the counterfactual cannot be observed or known with complete certainty, the concept of comparing observed outcomes to this hypothetical state underlies all valid approaches to assessing impacts. Valid comparisons imply that the net effect of interventions is isolated from all other extraneous or confounding factors that influence defined outcomes. Outcome Observed outcome with intervention } Impact What would have happened without intervention Time 26. The major challenge in impact assessments is to estimate the effect of programs after netting out extraneous confounding factors that affect outcomes. Threats to internal validity are generally grouped under several broad categories such as external events, secular drift, maturation, regression and attrition. 11 Another threat to the validity of claims concerning the impact of initiatives arises from the voluntary nature of programs. Some members of the target population may be more inclined to participate due to greater interest, motivation or other conditions. This self-selection process can bias results if the factors that lead organizations to participate are related to the specific outcomes under study. 9 See Mosteller and Tukey (1977). 10 This is sometimes referred to as additionality. 11 Readers interested in exploring these concepts in more detail are referred to Cook and Campbell (1979). 31

32 27. The strength of causal inferences that can be drawn from an analysis of particular intervention depends on how well the particular approach used in assessing impacts deals with the threats to validity. While there are numerous variations, impact assessment can be grouped under four basic headings based on the type of controls used to isolate program effects from other confounding factors experiments with random assignment, quasiexperiments with constructed controls, before-after studies with reflexive controls, and participant judgment and expert opinion. 12 Each of these is discussed below. Method Experiments with random assignment Quasi-experiments with constructed controls Before-after studies with reflexive controls Participant judgment and expert opinion Description Entities in the treatment group participate in the program; those in the control group receive alternative services or none at all. Random assignment helps guarantee that the two groups are similar; extraneous factors that influence outcomes are present in both groups. Because of this comparability, claims that differences between the two groups are the direct result of the program are more difficult to refute. Experimental designs are used quite frequently to test the efficacy of new treatments in health, social welfare and education. However, this approach is often difficult to implement Political considerations make random assignment difficult The services provided to entities may not be standardized It is frequently hard to maintain experimental conditions Experiments tend to be costly and difficult to administer The performance of participating entities is compared to other similar entities that have not participated. However, instead of random assignment, a comparison group is constructed after the fact. Valid comparisons require that the two groups be similar with respect to key characteristics, exposure to external events and trends, and propensity for program participation. To the extent that the two groups are similar, observed differences can be attributed to the program with a high degree of confidence. There are several types of designs that fall within this general category: regression discontinuity, statistically equated controls, matched controls, and generic controls. Many programs have been evaluated by comparing the performance of program participants before and after the intervention, attributing all of the change to the program. While widespread, results from studies that rely exclusively on reflexive controls should be treated with substantial skepticism. Before-and-after design cannot isolate the impact of the program from these extraneous factors such as selection bias, maturational trends, secular drift and interfering events. This approach relies on program participants or independent experts to make judgments concerning impacts. Individuals are asked to estimate the extent to which performance was enhanced as a direct result of the program in effect, to compare current performance to what would have happened in the absence of the program. This approach requires people to be able to determine the net effect of the intervention based solely on their own knowledge. It is often the only option available given data and budget constraints. When used, care should be taken to make sure that people consider the counterfactual in their assessment of impacts. 12 Case studies can also be used to examine the impacts of intervention. Unlike the other approaches, case studies rely on extensive narrative descriptions and other evidence to assert that the intervention caused observed outcomes. In general, case studies involve multiple sources of information including direct observation, interviews, documents, and physical artifacts. While case studies can provide rich explanations of how and why the program affected particular firms, it is difficult to generalize results beyond the firms studied. 32