Public Disclosure Authorized. Public Disclosure Authorized. Public Disclosure Authorized. Public Disclosure Authorized

Transcription

1 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Impact Assessment Report Fiscal Year

2 Introduction... 3 Methodology... 3 Figure 1: Impact Assessment Methodology... 4 Table 1. Overall Satisfaction... 5 Customer Satisfaction... 5 Table 2. Overall Satisfaction... 5 Figure 2. Staff vs. Consultants- Client Satisfaction... 6 Figure 3. Perceived Value of Training... 6 Impact Assessment: Training Services... 6 Figure 4. Skills Gained and put to use- Small and Medium Enterprises... 7 Figure 5. Changes in Company and Performance Improvement... 8 Table 3. Changes in Profitability... 8 Figure 6. Skills Gained and put to use- Business Development Service Providers... 9 Figure 7. New Products and Services- Business Development Service Providers... 9 Figure 8. Changes in Sales Figure 9. Skills Gained and put to use- Financial Institutions Figure 10. Changes in Company and Performance Improvement Figure 11. Loan/loss provisions and Loans affected by arrears Figure 12. Number and Volume of loans to SMEs Impact Assessment: Retail Consulting Figure 13. Positive Impact and Made at Least One Improvement Figure 14. Implemented Recommendations Figure 15. Change in Profitability- Retail Consulting Table 4. Other Effects of SEED s intervention Table 5. How Can SEED be Helpful to Your Business Going Forward?/Suggestions for the Future Impact Assessment: Business Enabling Environment Building the Market Figure 16. Would you use SEED services again? Figure 17. More likely to seek out external training and/or consulting services Figure 18. Demonstration Effects Conclusions Lessons Learned: Evolution of the Impact Assessment Process Next Steps

3 Introduction One of the cornerstones of the new SEED strategy is a focus on impact throughout the project development process. This means first, making impact a primary consideration in project design and implementation, and second, committing to follow-up with our clients after completing interventions in order to assess the results of our work. This report is a summary of the information gathered during the follow-up process for Fiscal Year Within SEED, the impact assessment process has three main purposes: a) institutional learning and development, b) evaluation of the efficacy of current programs and projects, and c) to provide donors, partners, other facilities, and staff with a report on the impact of SEED s activities. To accomplish these goals this report seeks to answer several basic questions: Were SEED s clients satisfied with the services that they received? At what level are SEED clients willing to pay for training and consulting services? Have SEED clients made changes in their operations as a result of SEED services? Have these changes translated into measurable impact on the profits or employment levels of SMEs? How has SEED affected the market for business services in the region? How has SEED contributed to improving the overall business environment in the target countries? The hope is that by answering these questions SEED will be able to improve on weaknesses, better utilize its strengths, and target its services more effectively in the future. Methodology The Metrics and Evaluation (M & E) is situated in the SEED s Sarajevo office and is not organizationally connected to any of the projects they are called upon to evaluate. SEED s impact assessment methodology is based on the Project Development Facility M&E Guide developed by the IFC, and extensively modified according to input from project managers, clients, and other impact assessment programs. The impact assessment process is outlined in figure 1 below. Each project manager is responsible for filling out a Project Completion Note (PCN) at the end of a project. The PCN highlights the expected outputs of the intervention, which is then targeted for impact assessment. Also at a project s end, the customer is asked to fill out a Customer Satisfaction Survey (CS), which is delivered to the M&E Team. For trainings, the CS is done at the end of the training. For consulting interventions, the CS form is left with the client with instructions on how to fax or return it to the M&E Team. This is sometimes followed-up with an interview by a member of the M & E Team. 3

4 Three months after a training intervention and at both six and twelve months after a consulting intervention is completed, impact assessment forms are sent to the recipient. Then a follow-up interview is conducted by a member of the M&E Team. These results are compiled, along with information from the PCN and the CS, to become part of the Project Monitoring Report (PMR). This document gathers most of the pertinent information about a project into one place for easy examination. The PMR provides a powerful management tool by which SEED can evaluate the quality of its deliverables, its overall impact on SMEs, and assess SME sector needs and demands. In additionally, management will be able to examine everything from individual consultants, program officers, and trainers up through entire product lines based on customer feedback and impact. This information enables managers to address specific client issues with individuals, make informed decisions when hiring consultants, as well as make broad decisions about future programming. Figure 1: Impact Assessment Methodology Project Completion Note Project Impact Assessment Project Monitoring Report Customer Satisfaction Survey Project Completion Note (PCN) is done by Project Managers Impact Assessment (IA) and Customer Satisfaction (CS) surveys are filled out by clients Project Monitoring Report (PMR) is prepared by the Metrics & Evaluation team The impact assessment process has been further tailored to fit other projects for which the above methodology would be either inappropriate or inadequate. For projects lasting longer than six months, such as work with business membership organizations, government ministries, or linkages clients, customer satisfaction surveys and impact data are collected every six months for the life of the project. For Business Enabling Environment (BEE) projects, which most often operate on a longer timeline and have less 4

5 concrete target clients, the impact assessment process is designed for the individual project by the M&E Team and is updated every twelve months. FY03 Process As this is the first year of SEED-wide formalized impact assessment, the process is continually evolving. The processes used, questions asked, and personnel involved changed during the course of the year as a result of trial and error as will be discussed below. In particular, the long time-lag between the end of some projects and the followup resulted in much more non-participation than was the case for projects later in the year. This should not be as significant a problem in the future. In the writing of this report on the FY03 impact assessment process, the M & E team utilized 543 Customer Satisfaction surveys, 368 Impact Assessment forms, out of a total potential pool of 1217 and 1213 total clients respectively. This is a very substantial response rate. In addition, the M & E team conducted a variety of interviews and surveys tailored to individual projects. Table 1. Overall Satisfaction Customer Satisfaction Impact Assessment Retail Consulting Retail Training Wholesale Financial Institutions Wholesale Business Development Services Total Total Clients in FY Response Rate 44.62% 30.34% Customer Satisfaction Customer satisfaction surveys are intended to assess client feelings about staff knowledge, listening skills, subject matter expertise, timeliness, consultants used, value of the service, and overall feelings about SEED services at the time of the intervention. As shown in table 1, SEED clients were extremely happy with the level of service they received. Over 95% indicated that they were satisfied or very satisfied with the trainings. Table 2. Overall Satisfaction N= 540 Overall Satisfaction % Very Satisfied % Satisfied % Neutral % Dissatisfied % Very dissatisfied %

6 This was consistent with positive reviews of the timeliness of service, and general impression of SEED staff. When trainings were not conducted by SEED staff, participants were marginally less satisfied with the quality of replacement consultants. The overall level of satisfaction with consultants was still over 9, but the relative number of those who answered Very Satisfied to those who answered Satisfied was significantly lower. Figure 2. Staff vs. Consultants- Client Satisfaction SEED Staff Consultant 0.0 Very Satisfied Satisfied In terms of perceived value for the services provided, the vast majority of SEED clients felt that the services were worth either the amount they paid or more than what they paid for them. This is significant progress along SEED s eventual goal of creating a paying market for its services, and demonstrates a strong demand for SEED s services. Figure 3. Perceived Value of Training Value of Training Higher than fees Value of training equivalent to the fee paid Training not worth the fee paid A more detailed breakdown of the customer satisfaction survey results for each type of intervention is available in Annexes A through D. Impact Assessment: Training Services SEED s training program is divided into three major pieces: Retail trainings, or trainings directly to Small and Medium Enterprises (SMEs); Business Development Services trainings, or trainings to companies, groups, and institutions who provide services to SMEs; and Financial Institution trainings, or trainings directly to banks, finance 6

7 companies, and others who might provide financing to SMEs. Impact assessment interviews are conducted by the M & E team several months after the event. The full survey results for each training type are available in Appendix E through H with detailed breakdowns. The following is a brief summary: Retail Training SEED s retail training efforts have focused on giving managers and executives in SMEs the skills they need to develop and expand their businesses. This includes using financial statements, managing information flow, marketing, human resources, sales, and strategic planning, among others. As illustrated by figure 4, as a result of SEED s training, retail training clients overwhelmingly feel that they have gained new skills they can put to use in their companies. Figure 4. Skills Gained and put to use- Small and Medium Enterprises 9 78% Gained New Skills Skills put to use Eighty percent of clients also felt that they made changes in their companies as a direct result of SEED trainings, and fifty-one percent felt that those changes led to an improvement in performance. Because only a limited amount of time passed between the training and the impact assessment, it is possible that performance improvements simply need more time to manifest. Nevertheless, the perception is still important. 7

8 Figure 5. Changes in Company and Performance Improvement 8 51% Made Changes in Company Performance Improved In terms of the bottom line, twenty-two percent of clients felt that their profitability had changes as a result of SEED s trainings. Table 3. Changes in Profitability Profitability changed? % of answered Better 22.52% Same 59.46% Worse 0.9 Do not know 17.12% As a whole, it is clear that SEED is having a positive impact on SMEs in terms of their business processes and the skill levels of their employees. What is unclear at this point is whether those improvements translate into increases in profitability or employment. A complete breakdown of our retail training impact assessment results is available in Appendix E. Business Development Services Trainings SEED s Business Development Training efforts have been of two types. The first has been developing and delivering courses on how to market, manage, and expand a consulting business, similar to the trainings given to other SMEs. The second has been a series of train-the-trainer courses designed to enable other consultants to conduct SEED courses using SEED materials. As illustrated by figure 6, Business Development training clients overwhelmingly feel that they have gained new skills as a result of SEED s training. Sixty-four percent felt that they were immediately able to put those skills to work in their company, and another twenty percent expected to be able to put their new skills to work in short order. 8

9 Figure 6. Skills Gained and put to use- Business Development Service Providers 96% 84% Gained New Skills Skills put to use or expect to put skills to use One of the major objectives of SEED s work with BDS providers is to enable them to develop a series of products that are aimed at SMEs, for which SMEs would be willing to pay. By these standards, SEED s BDS work seems to be very effective. Fifty-six percent of BDS training recipients report that their companies have introduced new products or services geared towards the SME market. Sixty-seven percent of those who introduced new services, say that they have been able to sell the new products. In general, forty-one percent of BDS training recipients say that their private sector base has increased as a direct result of SEED s trainings. Figure 7. New Products and Services- Business Development Service Providers 56% 67% 41% Introduced new products or services? Sold these new products or services? Increased your private sector base? Overall, SEED s BDS clients felt that, as a result of SEED s intervention, their company sales had either improved already or were expected to improve shortly. 9

10 Figure 8. Changes in Sales 42% Yes Not yet, but we expect them to be No 8% Sales higher than they would have been? It is clear that SEED is having a positive impact on Business Development Service providers, and the range of services available to SMEs in the region. A complete breakdown of our business development service training impact assessment results is available in Appendix F. Financial Institution Trainings SEED s interventions with financial institutions have largely been geared toward expanding the ability of SMEs to access finance. They have taken the form of a series of trainings for loan officers about how to read SME financial statements and business plans, and how to make credit decisions regarding SMEs. These trainings were generally arranged and paid for by the institution itself. The vast majority of SEED s financial institution clients felt that they gained new skills as a result of SEED s trainings, and sixty-nine percent said they were able to put those skills to use. Figure 9. Skills Gained and put to use- Financial Institutions 94% 69% Gained New Skills Skills put to use 10

11 Seventy percent of clients also felt that, as a direct result of SEED trainings, they made changes in their companies, and ninety-four percent felt that they had made at least one improvement. However, only thirty-one percent felt that those changes led to an overall improvement in the performance of the institution. Figure 10. Changes in Company and Performance Improvement 94% 69% 31% Made Changes in Company At least one improvement Overall Performance Improved Two measures of the effectiveness of our trainings with financial institutions suggested by SEED staff were the level of loan/loss provisions and write-offs at a financial institution and the number of loans affected by arrears. In both of those categories, the impact results were minimal: only twelve percent registered that SEED s services resulted in improvements in those categories. A set of more detailed questions is being incorporated into the impact process for Fiscal Year 2004 to explore this. Figure 11. Loan/loss provisions and Loans affected by arrears 12.5% 12.5% Better Same 87.5% 75. Worse 12.5% Loan/loss provisions and write offs Loans affected by arreaers Another important barometer of the efficacy of SEED training is whether the institution has increased the level of financing that it provides to SMEs as a result of SEED s 11

12 trainings. In this arena, there are significantly more positive results. Forty-seven percent of financial institution training clients report an increase in the volume of term loans to SMEs, while fifty-three percent report an increase in the number of SME borrowers. Figure 12. Number and Volume of loans to SMEs 47% 53% Increase in the volume of term loans to SMEs Increase in the number of SME borrowers It is clear that as a result of SEED services more financial institutions are expanding the volume and scope of their work with SMEs: A clear success. However, it is equally clear that a great deal remains to be done in this area. A complete breakdown of our financial institution training impact assessment results is available in Appendix G. Impact Assessment: Retail Consulting The impact assessment process for Retail Consulting clients was the most problematic during FY 03. There were several reasons for this. First, there was a relatively small pool of clients (31 eligible for Customer Satisfaction surveys and 27 eligible for Impact Assessment). Second, the clients who needed to be contacted were primarily the managers and/or CEOs of the businesses with which SEED worked, and they were often less than eager to discuss SEED s work with them or fill out forms. Third, at the beginning of the process our forms contained a number of very specific questions about profits, sales, turnover, and employment, which we intended to compare with baseline data gathered during our interventions with them. However, the companies were either unable or unwilling to provide us with updates about that information, effectively crippling our ability to assess impact. Because of the reluctance of companies to provide specific quantitative information, part of the way through the process we updated the forms to ask more qualitative questions about SEED s impact on the company. In this way, we could obtain a better sense of the company s perception of SEED s impact, although we would have only a limited amount of quantitative data with which to back up those perceptions. While in the future, we expect this approach to yield better results, for FY03, we have a very limited pool of data 12

13 from which to work. The full results of our analysis are available in Appendix H. For that reason, in this section we list the exact number of respondents to each question. All of the clients who answered the question, What kind of impact did SEED have on your business overall? (three of the five who were asked) responded that the impact was positive. Further, ninety percent of respondents (nine of the ten who were asked) listed at least one improvement made to their business as a result of SEED s services. Figure 13. Positive Impact and Made at Least One Improvement 9 Positive impact Made at least one improvement However, of the five institutions who responded to the question, If SEED made specific recommendations to you, have those recommendations been implemented? Two indicated that they had implemented them, while three dropped them. Figure 14. Implemented Recommendations 6 4 Implemented Recommendations Recommendations Dropped Of the eight institutions who responded to the question, How has your profitability changed as a result of SEED services? Three chose better, while the rest indicated no change. 13

14 Figure 15. Change in Profitability- Retail Consulting 38% 62% Better Same Worse Change in Profitability Included in all of our impact forms and as part of all of our impact interviews is a section in which we solicit general comments thoughts for going forward and suggestions for improvement about work with SEED. For this year, these comments were far more telling than the data itself. Below are the complete list of comments from retail consulting clients: Table 4. Other Effects of SEED s intervention Other effects - Comments It affected our knowledge about management and different financial sources. Increased motivation of employees and creating a winning atmosphere. Without SEED, there would be no foreign investor; the company would not have survived. We are more focused on costs and on marketing activities. In realizing where the company stands and where it should go. Improved our organizational structure. Cessation of overtime work Table 5. How Can SEED be Helpful to Your Business Going Forward?/Suggestions for the Future Forward assistance Comments/Suggestions Providing contacts of representatives and consultants of other markets. Partner in the future projects of our company. Monthly control of the business, and meeting with management. We are extremely pleased that SEED is promoting us through its various activities. We are interested in seeking out a partnership for new projects. Help in following the modern trends in business including management, technology etc. Concrete projects We would like SEED's help in developing better human resources. We need long-term financing and more affordable capital than can be found within this market. They should follow up more consistently. 14

15 The responses and comments seem to indicate that SMEs have made changes as a result of SEED s involvement with them, and that they see those changes as improvements. There is also clearly a call for more help from SEED and for a greater level of follow-up. It is unclear from the data available on SEED s interventions, what impact if any SEED s intervention is having on profitability. Impact Assessment: Business Enabling Environment SEED s work in improving the Business Enabling Environment has taken a number of forms, from contributing the draft laws, to conducting studies, to providing training, to organizing roundtables, and various others. Much of this work is by nature long-term, multi-faceted, and its impact is often difficult to quantify. For the purposes of Impact Assessment, SEED operates on the assumption that it is premature to look for measurable results in less than a year, and that each intervention requires a specific means of measuring impact. For this report, the M&E Team conducted impact assessment on BEE projects which ended before FY 03, the only projects which have fulfilled the one year qualification for BEE impact analysis. Appendix I contains a chart with a summary of the results. Overall, impact assessment was difficult to do for most of the pre-fy03 BEE projects. These projects were largely studies, publications, or advisory work, all of which have results which are very difficult to quantify. Some efforts were made to follow-up on the distribution lists for various publications, but these efforts largely did not bear fruit. In addition, many of the pre-fy03 BEE projects later became the basis for other work, and their results should be analyzed within that context. For example, the large amount of work SEED has done with leasing in all of the countries should be evaluated based on the impact of the legislative work which both took place during FY03 itself and continues to take place within the leasing platform. Largely, this year s efforts are intended to set the stage for the work that will begin in FY04. Appendix I contains a summary of the results for pre-fy 03 projects, while appendix J contains a list of projects slated for impact assessment in FY04. Building the Market One of SEED s major objectives is to build the market for paid consulting and training services in the region. SEED s work on the supply side of the market largely falls under its work with Business Development Service providers and SEED the development of the staff itself. From the demand side, SEED s work includes increasing the market both for SEED services and for the services of other consultants and trainers. The demand work is accomplished largely by demonstrating the benefits that can be achieved by obtaining high quality consulting and training services. In terms of developing the market for SEED services themselves, over ninety-seven percent of SEED s clients are interested in obtaining other services from SEED. Anecdotally this is supported by the large number of clients who have attended more than one of SEED s trainings, or who have become training clients after having received consulting services. 15

16 Figure 16. Would you use SEED services again? Business Development Services 99% Financial Institutions Retail Consulting 9 Yes Retail Training 99% In terms of developing the market for external consultants from non-seed providers, the vast majority of SEED clients indicate that they are more likely to seek out external training and consulting services after having worked with SEED. This constitutes a significant success in terms of creating demand in the region. Figure 17. More likely to seek out external training and/or consulting services 94% SEED-wide Yes, Training Yes, Consulting 89% A secondary impact of SEED s work is that successful interventions with a client can lead to other businesses being motivated to seek out similar services or to make changes in their business in response to changes made by our clients. As shown in Figure 18, the survey results appear to support this conclusion. 16

17 Figure 18. Demonstration Effects 73% SEED-wide Yes, Consulting/Training 38% Yes, Making changes in strategy/operations Conclusions Impact Assessment is necessarily an inexact process, and SEED s efforts to measure impact are still in the first stages. That said, some preliminary conclusions can be drawn based on the information gathered in this process, with the recognition that these conclusions will be subject to continual update and review as the impact assessment process continues. This report intended to answer several basic questions: Were SEED s clients satisfied with the services that they received? Yes, SEED s clients are generally very satisfied with the level of services they receive from SEED, in every type of intervention. They are pleased with the competence of SEED staff, and generally pleased with the abilities of the consultants with which SEED works. They overwhelmingly report that they would utilize SEED services again. At what level are SEED clients willing to pay for training and consulting services? SEED s clients are willing to pay for the services they received. The vast majority of clients are willing to continue to pay at the current level, and a significant minority would be willing to pay more. Have SEED clients made changes in their operations as a result of SEED services? SEED clients have made significant changes in their operations as a result of SEED services, and they feel those changes to have been an improvement. In the case of business development service companies, this includes an expansion in the number and types of services available to SMEs. In the case of financial institutions, this has included an expansion in the number and volume of loans given to SMEs. Has that impact translated into measurable impact on the profits or employment levels of SMEs? 17

18 As of yet, there is little evidence that SEED s interventions have led to an increase in the profits or employment levels for SMEs. There are also mixed results about whether SEED s intervention led to an overall improvement in the performance of the company. In many cases, further follow-up will be needed as not enough time has passed for the changes implemented to be translated into improvements in the bottom line. How has SEED affected the market for business services in the region? The results indicate that SEED has contributed to the development of the business service market, both by increasing the supply of services available to SMEs, and by increasing the demand for those services. It was also seen that SEED services have a demonstration effect beyond SEED s clients. How has SEED contributed to improving the overall business environment in the target countries? It is too early to offer any significant results from SEED s BEE work, although early indications are positive. SEED will continue to monitor the progress of these projects. Lessons Learned: Evolution of the Impact Assessment Process During the course of the FY 03 Impact Assessment process, the Metrics and Evaluation team and SEED in general learned a great deal, largely by trial and error. As a result of this learning process, several changes have been made to the impact assessment process. Most of these changes were operational in nature and involved what questions to ask, in what form to ask them, and how to go about organizing the process. There were a few more systematic changes that will be incorporated into the process going forward. In general, impact assessment will be more closely integrated into SEED s business processes. This involves thinking about impact in every stage of product and project development. Sections on impact assessment have been added to every platform and project document, and project managers will be expected to put thought into how the impact from a project will be measured before a project begins. Impact assessment is also going to be part of every partnership agreement SEED signs in the future. This will contractually obligate companies to participate in our impact process, and should help us to get a better return, particularly from Retail consulting clients. As per the new strategy, our interventions will be of a longer duration and a greater depth in general, which should help us to maintain the relationships so essential to impact assessment. In addition, working through business membership organizations will enable us to utilize those connections in the impact assessment process. Now that the impact assessment process has been regularlized into SEED s operations, SEED will be able to be more timely in following up with clients. 18

19 Certain issues require a greater depth of information in order to be operationally useful. Based on the FY03 results, the FY04 process will pursue a greater level of information regarding several issues. Continuing improvements in Project Tracking and Management Reporting System, SEEDS management information software, should make the impact assessment process much easier to manage, and more useful for country and project managers. Next Steps It is SEED s intention that this document is merely the first step in a long process of integrating the results of the impact assessment process into our programming going forward. The report is merely the starting point for ongoing assessments, adaptation, and program development. This information is useless unless it is operationalized at every level, from the CEO to each project manager, and it is SEED s intention to see that impact assessment becomes an essential part of our businesses process. 19