Measuring Performance in Business Enabling Environment Projects IFC Monitoring and Evaluation Team* Cairo, Egypt November 29- December 1, 2005 * Geeta Batra Mark Bardini Benjamin Reno Weber
I. Introduction As the scope of IFC s Technical Assistance (TA) work has broadened, projects to improve the Business Enabling Environment (BEE) in our target countries have become an increasing part of IFC s portfolio. This work has enormous potential impact, both in terms of the stability of IFC s investment portfolio and on the productivity of businesses at large within IFC s target countries. However, measuring the impact of this work presents unique challenges. There has been a great deal of discussion about what exactly is BEE? For the purposes of our work, we define BEE as projects designed to improve institutional legal and regulatory conditions under which businesses operate, including changes to government policies, laws, regulatory frameworks and administrative practices by public sector entities. The range of issues encompassed in this definition is wide and includes several stakeholders both within and outside of the World Bank Group. To date, the IFC has worked in a number of key business enabling environment areas essential for long-term development of the private sector. For example, in the financial markets, IFC has worked to improve the regulation of banks and capital markets, reformed bankruptcy procedures, introduced leasing, factoring, and microfinance, improved corporate governance procedures, and privatization laws. Within the tax system, IFC has worked to streamline administrative systems and introduce business-friendly legislation. IFC has worked to improve contract enforcement and introduce alternative dispute resolution into the judicial system. To improve the market for businesses generally, IFC has helped countries to reform their trade policies, their customs administration, their registration and licensing procedures, their inspection regime, and procurement policy. Working directly with local, regional, and national governments, IFC has helped them to improve their transparency, accountability, and overall effectiveness. IFC is committed to strengthening the design and implementation of its Technical assistance in client countries. To accomplish that goal, IFC is in the process of implementing a comprehensive monitoring and evaluation system, and draw on these results to inform management.. An important step would be to track and monitor relevant, meaningful, cost-effective performance indicators to measure the outputs, outcomes, and impact of TA projects. This paper summarizes IFC s recent efforts in measuring the performance of BEE projects, identifies some of the the challenges going forward, and discusses IFC s approach in addressing some of these challenges.
II. Measuring Performance in Projects with BEE Components The main challenge in measuring the impact of BEE programs is that these projects or projects with BEE components, by their nature affect a wide pool of end-recipients, many if not most of whom will not know what organizations were involved or in many cases be able to identify precisely how the project impacted their business. This makes it difficult to determine impact without broad surveys and/or detailed economic indicators not easily available in may of IFC s host countries. Even if this is possible, the costs of obtaining that information, particularly in developing countries where data is difficult to obtain, would be a challenge. Keeping the above constraint in mind, IFC has adopted the program logic model to define the objectives and measure the performance of its BEE programs. This approach is consistent with the logical framework method applied by several donor and international agencies. A. The Framework :Developing the Logic Model Figure 1 below shows the logical framework model in general. This logical framework lays out all of the envisioned activities/outputs, intended outcomes, and targeted impacts and gives examples of each. Figure 1: Program Logic Model Outputs Outcomes Impacts Products of Program Activities: Advisory Services to Gov t Public Awareness Campaign Laws Drafted Changes in knowledge, behavior and performance of beneficiaries: Gov t entity reporting increased skills SMEs reporting improved knowledge Legislation adopted Benefits accruing to target population: Improved services to SMEs SMEs utilizing available services Fewer barriers for businesses
Figure 2 shows the Program Logic Model for IFC s BEE projects. Figure 2: BEE Program Logic Model Outputs Outcomes Impacts Outside boundaries of accountability B. Developing and Collecting Information Based upon these logical frameworks, indicators are developed for each stage of the project. These indicators should be relevant, universal, reliable, and relatively easy to collect. They should be applicable to the program being monitored, but general enough to offer benchmarking with similar projects in other parts of the world. They should also utilize a methodology that will yield consistent results, and the cost of collecting the information should be economical.
Some of the output and outcome indicators that will be collected as core indicators for BEE projects are shown in tables 1 and 2. These lists are by no means exhaustive and all encompassing, and the list would be expanded based on specific project objectives. The intention of this exercise is to be able to aggregate across different projects and compare across geographic regions. Table 1: Output Indicators of BEE Projects Activity Indicator Data Source Assessments Number of reports Program Records Advisory Services Number of advisory service hours provided Program Records by IFC and third-parties under contract to IFC (by target group) Number of laws/regulations drafted or Program Records contributed to the drafting Number of unique organizations to which Program Records advisory services were provided (by target group) Training Number of training courses provided Program Records Number of training participants (by target Program Records group) Number of participant-training hours Program Records provided (by target group) Number of unique organizations that sent Program Records individuals for training (by target group) Public Awareness Campaign Number of press releases distributed by the Program Records facility Number of TV/radio spots aired Program Records Number of unique visitors to Web site Program Records Number of public information events held Program Records Table 2: Outcome Indicators of BEE Projects Construct Indicator Data Source Population Outcomes Changes in the law and regulations related to business enabling environment Changes in relevant administrative procedures Transaction cost for specific procedures Days required to complete specific procedures Yes/No (based on standard checklist) Yes/No (based on standard checklist) Total cost (excluding unofficial payments) born by company to complete specific procedures, including fees to government and third-parties (e.g., attorneys) Time (calendar days) Program records Program records Doing Business SME survey Doing Business SME survey SMEs that completed procedures in the last 12 months SMEs that completed procedures in the last 12 months
Corruption Policy advocacy by business associations Sustainable business associations Value of unofficial payments to Doing Business public officials SME survey SMEs that completed procedures in the last 12 months Yes/No (based on standard Business Participating business checklist) Association associations Number of dues-paying members Percentage of BA total costs covered by fees paid by businesses survey Business Association survey Business Association survey Participating business associations Participating business associations. III. Other Experimental Approaches to Evaluation of BEE Projects IFC is piloting randomized control designs in a few countries such as Peru and Honduras For example, IFC LAC Facility and Lima Municipality have joined forces to simplify the business licensing process in Lima Municipality. Since September 2004 a joint-team of IFC staff and municipality officers has worked together to streamline the process of obtaining a municipal business license. As of 2004, new businesses were required to make a minimum of eleven trips to the registration office, to undergo on average of four inspections, and to deal with several different agencies within the Municipal office. The minimum time to complete this process was 35 days, the average time about three to four months. Under the new licensing process, businesses only need one inspection, the minimum time for completion is two days, and applicants deal with a single person within the Municipality. A pure randomized design would be ideal, but practical considerations forced us to modify the process. The main advantage of a randomized evaluation is to rule out selection bias. A randomized experiment of the registration process requires randomly assigning firms to either treatment (new process) or control group (old process). But the Municipality was legally prohibited from operating both processes simultaneously. For that reason, we implemented a modified approach, an experimental approach looking at results before and after the implementation. The main concern with this methodology is that the results could suffer from selection bias due to seasonal variation or time-specific shocks in the businesses applying for a license and due to differential self-selection before and after the switching date. In order to address these issues, firms are sampled as closely as possible to the switching deadline for the new process. This sampling strategy ensures that there is little change in the pool of applicants. 1 Second, the evaluation includes an additional control group of firms that finish the licensing process three months prior to the switching date. This control group 1 The Municipality plans to start advertising the modifications in registration procedures in November 2005. Assuming there is little publicity about the new licensing process before November, the project team does not expect a significant change in the type of businesses that apply for a license.
will be used to assess any seasonal or time-specific variation and to detect any changes in the registration process prior to the official switching date. 2 We are currently implementing this and will have results within the next 4 months. IV. Challenges Going Forward We have clearly made substantial progress in our monitoring and evaluation efforts. But we can always do more. Some of the challenges we face going forward are: (a) Data collection in developing countries data are not always easily available and can be very expensive (b) Aggregating performance indicators across the diverse projects and regional facilities to be able to provide a consolidated picture of our BEE work (c ) Implementing more experimental designs to be able to address the issue of attribution 2 The Municipality started to implement minor changes to the registration process as early as March 2005. We will compare the result from the two control groups with the results from the initial diagnostic report to ensure that none of these changes had a significant impact on the registration process.