FOR OFFICIAL USE ONLY

Transcription

1 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Document of The World Bank FOR OFFICIAL USE ONLY IMPLEMENTATION COMPLETION REPORT Human Development Sector Unit East Asia and Pacific Region INDONESIA SKILLS DEVELOPMENT PROJECT (Loan No IND) June 30, 1999 Report No This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization

2 CURRENCY EQUIVALENTS Currency Unit = Indonesian Rupiah (Rp) US$1.00. Rp 2,105 in SAR (February 14, 1994) US$1.00 = Rp 2,400 (July 1994 to July 1997) US$1.00 = Rp. 5,000 (July 1997 to July 1998) FISCAL YEAR April 1 - March 31 ABBREVIATIONS AND ACRONYMS APINDO - Indonesian Employers' Association BAPPENAS - National Development Planning Board CAS - Country Assistance Strategy CGTUs - Company Group Training Units CPIU - Central Project Implementation Unit GOI - Government of Indonesia I BRD - International Bank for Reconstruction and Development ICR - Implementation Completion Report ITS - Industry Training Scheme KADIN - Indonesian Chamber of Commerce and Industry KPKN - State Treasurer LLMIS - Local Labor Market Information Systems MOM - Ministry of Manpower MTM - Medium Term Models OECD - Organization for Economic Cooperation and Development PIMPRO - Project Director PIU - Project Implementation Unit PMP - Young Professional Entrepreneur Development Program PSC - Project Steering Committee PTF - Provincial Training Fund SAR - Staff Appraisal Report SDP - Skills Development Project SETAS - Small Enterprise Training and Advisory Scheme SOP - Standard Operating Procedures SPABP - Fund channelling mechanism from national to provincial level STM - Short Term Models TOR - Terms of Reference Vice President Sector Director Country Director Task Manager Jean-Michel Severino Alan Ruby Mark Baird Brigitte Duces

3 FOR OFFICIAL USE ONLY TABLE OF CONTENTS PREFACE... EVALUATION SUMMARY... i ii PART I: PROJECT IMPLEMENTATION ASSESSMENT...1 A. Statement/Evaluation of Objectives... 1 B. Achievement of Project Objectives... 3 C. Major Factors Affecting the Project... 7 D. Project Sustainability... 9 E. Bank Performance F. Borrower Performance... 1 I G. Assessment of Outcome H. Future Operations I. Lessons Learned PART II: STATISTICAL TABLES Table 1: Summary of Assessments Table 2: Related Bank Loans/Credits Table 3: Project Timetable Table 4: Loan/Credit Disbursement: Cumulative Estimate and Actual Table 5: Key Indicators for Project Implementation Table 6a: Quantitative Indicators for the Operation of the Provincial Component Table 6b: Quantitative Indicators for the Operation of the Ministry of Manpower Component Table 7: Studies Included in the Project Table 8a: Project Costs Table 8b: Project Financing Table 9: Status of legal Covenants Table 10: Compliance with Operational Manual Statements Table I1: Bank Resources: Staff Inputs Table 12: Bank Resources: Missions ANNEX A: BORROWER'S CONTRIBUTION TO THE ICR ANNEX B: ICR MISSION'S AIDE MEMOIRE This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization.

4

5 -1i- IMPLEMENTATION COMPLETION REPORT INDONESIA SKILLS DEVELOPMENT PROJECT LOAN NO IND Preface This is the Implementation Completion Report (ICR) for the Skills Development Project in Indonesia, for which Loan No IND in the amount of US$27.70 million was approved on March 25, 1994, and made effective on June 13, Of the original amount, US$9,714,000 million was cancelled (US$4,914,000 on July 23, 1997 and $4,800,000 on August 26, 1998 respectively). The Loan was closed on December 31, 1998 after a one year extension. Of the original loan amount, US$15.42 million was disbursed, and a further US$2.56 million will be cancelled. The last disbursement took place on May 5, The ICR was prepared by Eamonn Darcy (Consultant) for Brigitte Duces, Task Manager, Education Sector, East Asia and Pacific Region (EAP) and reviewed by Alan Ruby, Benjamin Fisher and Oscar de Bruyn Kops. The peer reviewer was John Middleton. This report is based on material in the project file and ICR mission that was conducted in Indonesia during March 14 to 26, The Borrower prepared a contribution to the ICR that is attached as Annex 2. The Borrower also commented on the Bank's draft ICR.

6

7 - ii - IMPLEMENTATION COMPLETION REPORT INDONESIA SKILLS DEVELOPMENT PROJECT (LOAN NO IND) EVALUATION SUMMARY A. Introduction (i) The Skills Development Project (SDP) was the thirtieth project in the education and training sector to be financed by the International Bank for Reconstruction and Development (IBRD) in the Republic of Indonesia. Nineteen of these were implemented through the Ministry of Education and Culture, two through the Ministry of Manpower (MOM) and nine through other ministries. This demonstration project became effective on June 13, 1994 and with a one year extension, was closed on December 31, The project implementation period was extended (a) to provide more time for effective operation (which was delayed for almost two years out of a planned three) to demonstrate the potential usefulness of the private sector, demand-led training fund approach used in the project; and (b) to allow achievement of project objectives for a reduced number of activities namely: operation of the Provincial Training Funds; and in MOM, a second phase of the development of the Local Labor Market Information System, continuation of staff training, and evaluation of the project. B. Project Objectives (ii) The project's objectives were consistent with: (a) the Borrower's strategy for skills development; and (b) the Bank's strategy for human resources development in Indonesia as articulated in the country assistance. They are still relevant for the country's overall development. Specifically, the project's objectives were to: (a) develop and implement innovative industrial skills training schemes within private enterprises in three provinces (West Java, East Java and North Sumatra) for possible replication in other parts of the country in the future, as part of an overall training program to promote productivity and efficiency of the economy; (b) support a special training program for unemployed university graduates (PMP) in five provinces (the three mentioned above plus South Sulawesi and Bali); and (c) strengthen policy making, research, planning, and management capabilities of the Ministry of Manpower. (iii) The key agreements reached to promote the achievement of project objectives included: (a) the provincial training fund activities would be carried out in accordance with operating procedures and eligibility criteria acceptable to the Bank; (b) a guaranteed loan repayment scheme would be established using government funds for eligible people who completed the training program for unemployed university graduates; and (c) evaluation studies would be carried out not later than December 31, 1995 and December 31, C. Implementation Experience and Results (iv) The total project cost for the SDP was to be US$39.75 million of which the IBRD loan was to finance US$27.70 million and private companies and the Government of Indonesia were to finance US$12.00 million equivalent. Of the original loan amount, US$9,714,000 million was cancelled

8 (US$4,914,000 on July 23, 1998 and US$4,800,000 on August 26, 1998 respectively). The amount disbursed from the Loan was, after conversion, US$15.42 million. The last disbursement was on May 5, It is anticipated that a further US$2,563, will be cancelled. A key element of thedesign of the project was the contribution by companies to the cost of training on an upward scale of 20%, 40% and 70% per year of participation in the project. In practice, up to the 40% level was widely achieved but there was resistance to the 70% level that may have been partly caused by the economic downturn, though there was a high level of achievement of this level in East Java province. There was also resistance to the contribution by small companies as they normally qualify for full subsidisation of training costs under government programs. The payments were made in cash or inkind in the three provinces with the proportion differing between them. (v) Project design and preparation. The project design and preparation teams recognised the difficulty of implementing a decentralised project in three provinces where the administrative procedures involved were new to the central and provincial government officials, and to the private sector staff involved. For this reason, money from a previous project (Loan 2705-IND) was used to finance teams in the three provinces to prepare their subcomponents. However, in hindsight, the administrative procedures that were prepared were not comprehensive or detailed enough to facilitate implementation. A practicable set of standard operating procedures was only used during the last eighteen months of the four year project. (vi) Project financing issues. The mechanisms and detailed procedures for the transfer of loan funds from the national to the provincial level (SPABP), release of provincial government funds, and payments for services by the Provincial Government project manager (PIMPRO) were prepared by the Borrower. Again, as these were new to the numerous central and provincial level officials involved they had to be further detailed during project implementation to be effective. These administrative deficiencies led to serious problems that impeded the implementation of the Provincial Component during the first two years after the project became effective because there were delays of eight to ten months in the provision of project funds to the three provinces. These problems were overcome during the last two years of the project. These start-up problems caused temporary but not lasting damage to the implementation of the project. Disbursements were low and training and other project activities had to delayed during this stage of the project. (vii) A number of unacceptable practices were mentioned to the ICR mission; this information was provided verbally without written documentation or proof. In one province, training providers were asked to refund about 3% of the fees paid to them from the Loan to the PIU for services provided by the latter. About 1% to 3% of the provincial government funds provided to the project were repaid to them for services provided to the project in the three provinces. It was also stated that for the same reasons about 1% of the loan funds provided through the Treasury Office (KPKN) in the three provinces were repaid (although this was not independently verified with KPKN). (viii) Award of training contracts. There were also serious bureaucratic problems in the Provincial Govermments that caused delays in the payment of training providers that were only overcome during the third year of the project following intervention from the Bank. The Bank found that a small number of private training providers were winning a high proportion of contracts in the three project provinces during the first three years of the project, which was satisfactorily overcome in the extension phase due to action initiated by the Bank. One reason is that many firms would not tender because of the delay in being paid. The major, unacceptable reason was nepotism and there were serious examples of this in all three provinces. As a result of concern expressed by the Bank, staff

9 - iv - changes were made in the three provinces. It is not possible to give an objective view of the effect of these 'unacceptable practices' on the performance of the project as most of them were not disclosed until the end of the project (para. x). (ix) Project achievements. Based on the achievement of the key and quantitative indicators (Tables 5, 6 (a) and 6 (b)), the project's outcome was satisfactory for the achievement of implementation progress and development objectives for the last eighteen and twenty four months respectively, of implementation. The project was rated as unsatisfactory before this. The efforts made by the Borrower and the Bank during the first two years to overcome administrative obstacles and the adoption of standard operating procedures enabled the project to perform satisfactorily for about the final two years. The improved performance was assisted by the restructuring of the project in August Satisfactory, qualitative outcomes about the beneficial effects of private sector training and its quality were obtained from the evaluation studies but no relevant quantitative conclusions could be drawn from them because the path analysis analytical technique used expressed all conclusions as relationships between dependent and independent variables. The design of the project was shown to be practicable: decentralised, private sector dominated training funds subsidised training needs analysis and training of employees. The importance and value of training in firms was promoted. (x) During the first two years there were many administrative issues that impeded implementation of the provincial component and the Bank's supervision efforts were focused on the resolution of these. Because of this, it was not until 1996 that the conflict of interest in the award of training contracts and irregularities in the contracting process were detected by the Bank and subsequently acted on by the Borrower. While the Bank asked the Borrower to take corrective action soon after discovering the conflict of interest, the Bank could have put more pressure on to ensure that the necessary action was taken quickly enough by the Borrower. Project financing problems were not detected earlier because the provincial PIUs only began to talk about them during the final supervision mission in August 1998 and much more during the ICR mission in March Considering all of the issues that arose and given that one aim of this pilot project was to learn lessons from the experience of different approaches, overall Bank performance in project supervision was mainly satisfactory, despite some weaknesses, for this innovative project. (xi) The national level Central PIU (CPIU) and the Project Steering Committee (PSC) were set up to facilitate overall implementation and to take responsibility for policycoordination. The CPIU and the PSC should have played a more active role in the facilitation and monitoring of project implementation. For example, a more timely development of the standard operating procedures (SOP) would have given provincial authorities a firm administrative framework within which to work and would have reduced the scope for conflict of interest in the award of training contracts (para. viii). Borrower performance improved over time and was satisfactory in the second half of the implementation period. It was weak during the first half of implementation and in hindsight, also during preparation. D. Summary of Findings, Future Operations and Key Lessons Learned (xii) It is encouraging that despite resistance from some employers, many enterprises were willing to join substantially with the Government in managing and investing in the training of their workers, despite poor program administration by the Government in the first half of this project. The medium term implications for public training policy that the project model and incentives work are substantial. Valuable experience has been gained in the implementation of decentralised private sector training in three provinces.

10 (xiii) It seems that there was a large amount of money circulating in the project that was handled informally (written accounts were not kept) such as private sector contributions and the various refunds. The amount of money is unknown as (a) proper financial control was not kept over most of it, and (b) it has not been included in the audits. Some of the reasons for these informnal practices were valid, but others were not. The valid practices should have been formalised by modifying the operating procedures to include them, which was not done. There was financial control of the Loan and Government funds but these measures would have ensured financial control of the private sector funds. (xiv) Funds are being provided to sustain the Staff Training and Development, and Local Labor Market Information System activities in MOM. The Government maintains a high commitment to education and training and to the system to support in-service training in small and medium enterprises. The Ministry has stated that it wishes to provide funds to sustain the SDP program in a modified form in the three provinces from June 1999 and to replicate the prograrn in other provinces in the following year. In the three provinces, employer organizations and provincial government support a continuation of the project programs in various modified forms. In the light of the budgetary uncertainty and the very constrained economic environment in Indonesia at present, the sustainability of the Provincial Component has to be rated as uncertain in the short-term though it seems that it may be supported when the economy improves. (xv) In the light of the very constrained economic environment currently in Indonesia the Borrower and Bank have agreed priority for human resource development support would be given to maintaining enrollments for basic education. When the Indonesian economy has improved, future potential investment in the training subsector would be guided by the Sector Study 'Training and the Labor Market in Indonesia: Productivity Gains and Employment Growth' (Report No IND) and the lessons learned from this project. (xvi) There have been valuable lessons learned from this project and the main ones are: first, after the administration difficulties were overcome the institutional arrangements and the funding mechanisms used in the project were found to be suitable for the decentralisation of the management and delivery of training at provincial level; second, many private sector employers especially medium and large ones, are willing to take responsibility for planning and organizing the training of their employees; third, though there is some resistance especially from small firms, many enterprises accept the concept of cost sharing and it is estimated that in the present recession many medium and large companies would be willing to contribute an average of 40% to 50% of their costs of training. The experience gained by MOM under the project was valuable as under a new policy initiative, the Ministry has extended the use of variations of the cost sharing model to others of their labor market programs together with decentralized budgeting and control to provincial level; fourth, it took two years to get the special fund channelling mechanism (SPABP) used for transferring Loan funds from central to provincial level to operate smoothly as it was new to MOM and these provinces. As decentralization continues, this mechanism may have to be used more widely and therefore, in projects where it will be used care should be taken to ensure that detailed operating procedures are pilot tested during project preparation; fifth, if this project is to be replicated or related structures established that delegate management responsibility to provincial level, it would be essential that comprehensive and detailed operating procedures would be developed and approved during project preparation. These ex ante regulations would permit more ex post accountability for performance and use of funds, which should be carried out. Also effective central coordination and monitoring of project preparation and implementation at central level by the Borrower are essential; sixth, the original time-scale of three years would have been right for this demonstration project to test the mechanisms used in the absence of serious administrative delays. However, based on the

11 - vi - experience of the contributions from the private sector, providing subsidies for three years is too short a time to prepare enterprises to fully fund the cost of planning and delivering the training of their own employees, about six years would be required; and seventh, the evaluation studies were canied out at the end of implementation which allowed no opportunity for corrective action to be taken about the analytical method used. A pilot test of the methodology carried out earlier in the project would have highlighted the problem with the analysis and allowed time to correct it.

12

13 IMPLEMENTATION COMPLETION REPORT INDONESIA SKILLS DEVELOPMENT PROJECT (LOAN NO IND) PART I: PROJECT IMPLEMENTATION ASSESSMENT A. Statement/Evaluation of Objectives 1. Background. The project was designed as a three year demonstration project that built on previous projects in the training subsector. It set out to demonstrate the benefits of demand-driven and industry-led skills training programs through the provision of financial incentives to private enterprises that were willing and capable of establishing such training. The project represented a major shift from public to private sector provision of training in Indonesia and it included a significant decentralization of funding and operational responsibility from the national to the provincial level. 2. The Project Objectives and Components. The objectives as stated in the Staff Appraisal Report (SAR), Report No IND, were: (a) to develop and implement innovative industrial skills training schemes within private enterprises in three provinces (West Java, East Java and North Sumatra) for possible replication in other parts of the country in the future, as part of an overall training program to promote productivity and efficiency of the economy; (b) to support'a special training program for unemployed university graduates in five provinces (the three mentioned above plus South Sulawesi and Bali); and (c) to strengthen policy making, research, planning, and management capabilities of the Ministry of Manpower (MOM). 3. In order to achieve the objectives, the project consisted of two components: The Provincial Component included the establishment of a Provincial Training Fund (PTF) to implement four subcomponents in the three participating provinces: (a) start-up new training programs and activities in firms by establishing company group training units (CGTUs); (b) start-up new training programs and activities in individual companies by supporting them to employ new training managers, and provide limited staff and expenses to support employer associations (KADIN/APINDO) to promote industry-based training, called Industry Training Scheme (ITS); (c) support targeted training programs to train job-seekers for identified vacancies with specified placement targets in jobs related to the training; and (d) establish pilot programns to provide training and consultancy services to small-scale industries (SETAS). The Ministry of Manpower Component included four subcomponents: (a) establish programs to train and develop young professional entrepreneurs for self employment (PMP), primarily unemployed university graduates, in five provinces; (b) improve the local labor market information system; (c) support a staff development program to upgrade the capacity for

14 -2 - management, research, planning, and policy development; and (d) prepare a Provincial Training Fund scheme for South Sulawesi. 4. Cost Sharing. Cost sharing was to be a key element in the provincial component financed through the PTF scheme. The objective was to ensure that industry would finance in-service training after the project was completed. The proportion of costs financed from the loan was to be 80% in year one, 60% in year two and 30% in year three for the CGTUs, inservice training for large and medium firms, and for the training advisory services for small firms. Employment of training managers in firms was to be supported at 50% for two years to provide enough time to demonstrate their benefit to the companies that would then continue to bear their full costs. Training for job-seekers and small firms was to be 100% funded, the former being subject to placement performance. The PMP, a MOM subcomponent, was to be 100% funded, but participants were to contribute a specified amount of their own money to help capitalise their firms. 5. The loan financed 69.8% of total project costs and the private sector and the Borrower provided the balance. More than half of the loan (57%) financed training programs and training management and advisory services for companies and job-seekers under the provincial component. Under the MOM component, 13% of the loan financed fellowships and training. National and international consultant's services and studies were financed by 19% of the Loan while the balance of the loan financed equipment and unallocated items. 6. Evaluation of Objectives. The project objectives were consistent with the Bank's strategy for human resources development in Indonesia contained in the Country Assistance Strategy (CAS) presented to the Executive Directors in December This CAS recognised that investing in people would continue to be a critical ingredient for development, improving human welfare, slowing the rate of population growth, and enhancing the skills, productivity and employment prospects of the labor force. 7. The project's objectives were consistent with the Borrower's strategy for skills development and remain relevant at this time for the country's overall development. Within the MOM the low internal and external efficiency of its vocational training centres was correctly identified and the Ministry decided during project preparation not to invest in these centres until studies were conducted and a new strategy formulated. The Governrent supported the position that more demand-driven, industry-based training systems be developed and decided to provide the private sector with incentives to develop skills training capacity at the initial stage on a cost sharing basis. 8. As this was a demonstration project to test a new policy approach to private sector inservice training, careful attention was paid to project monitoring and evaluation in the SAR. Detailed quantitative, qualitative and process indicators were developed for each subcomponent (Tables 5, 6(a) and 6(b)). 9. In order to measure project impact, evaluation/impact studies of the CGTU and individual company training services, small enterprise training and advisory services, and the PMP were to be contracted by the MOM PIU. Two such studies were to be carried out at the end of 1995 and 1996 (Table 7).

15 Change in Project Design. The project as originally designed was a new policy direction for the borrower in this sector. A mechanism (SPABP) used by some other ministries but new to MOM and the Provincial Governments concerned was agreed for the transfer of Loan funds from the national to the provincial level. This mechanism only began to operate smoothly in the third and final year of the original phase and thus there were serious difficulties in implementation of the provincial component. There were serious bureaucratic delays at the provincial level that delayed the payment of training providers that were only resolved at the end of the original phase. For these reasons it was mutually agreed between the Bank and the Borrower that project management efforts would be better concentrated on the three original provinces and therefore, it was decided not to proceed with the preparation of a PTF scheme for South Sulawesi. 11. A key part of the design of the PMP was to be the provision of credit/financing to the participants in the program. MOM made determined efforts to get approval for and to fund a loan repayment scheme but was unsuccessful in this because of regulatory constraints. The Bank and the Borrower agreed to amend the Loan Agreement to delete this scheme in September Because of this reason and a delay in recruiting an international consultant, the PMP subcomponent was substantially delayed in starting up. Under the project there were to be short-term and medium-term training models. As a result of the delay, MOM unilaterally decided to use government funds to implement the short-term training model independently of the project and to extend coverage to twenty five provinces instead of five under the project. 12. At the request of the Government, the Bank agreed on August 19, 1997 to (a) a one year extension of the project, (b) a reallocation of funds across Loan categories, and (c) to cancel US$4,914,000 out of the original US$27.7 million Loan. The project was restructured at this time to continue with a reduced number of activities namely: operation of the provincial training funds; and in MOM, a second phase of the development of the Local Labor Market Information System, continuation of staff training, and evaluation of the project. B. Achievement of Project Objectives 13. The project's objectives as described in para. 2 were quite specific and quantitative; qualitative and process indicators were also established to measure achievement. Two studies were carried out at the end of the project to evaluate the provincial component: a record of experiences and an impact study as opposed to the two impact studies included in the SAR. The assessment here is based on measurement against the indicators and on the limited conclusions that can be legitimately drawn from the evaluation studies carried out. Table 5 assesses achievement of qualitative indicators while Tables 6(a) and 6(b) provide details of the achievement of the quantitative targets (paras. 27, 28 and 52 for further outcomes). Provincial Component Achievement of Quantitative Targets as set out in the SAR 14. Establishment of Company Group Training Units. The targets for the establishment of CGTUs for medium sized enterprises was substantially achieved in the original phase, as the

16 -4 - average number of companies in each CGTU was about 80% of the anticipated level. However, with sustainability in mind, there was a consolidation of CGTUs in the extension phase based on the criterion of quality of service, to about 50% of the number formed in the original phase. The CGTU concept has been supported by many medium sized enterprises that cannot afford a training function of their own. Other companies that originally joined CGTUs opted out later, mainly as they perceived training to be an extra cost that they could not afford or did not need at that time due to the economic crisis. 15. Training Provided to Medium and Large Enterprises. The actual outcome was an average of 85% of the target in the SAR for the three provinces. The training programs were designed in response to a training needs analysis and were generally of short duration of one to two weeks. They were generally for supervisory and higher grades of staff and were mainly focused on core, quality, personal and management skills. These activities were the main ones of the component (para. 20 for quality of training). 16. Training Provided to Small Enterprises. The training programs were for owners/managers of small enterprises in the formal sector. The actual outcome was an achievement of 102% of the target. The services provided were: (a) short training programmes generally for production, marketing and financial management skills, and (b) short tailored consultancy services focused on business planning and development for individual enterprises. 17. Job-Seeker Training. The objective of this was to demonstrate private provision of training for job-seekers with the payment for the service depending on the successful placement of trainees in jobs. It was a small-scale activity whose outcome was 99% of the target. Almost all trainees were placed in jobs. The training programs were three months in duration and were mainly for production operation skills. Record of Experiences and Impact of the Project 18. Evaluation of the Project. Two inter-related studies were carried out for this component (a) to record the experiences of the project and (b) to measure the impact of the project. The two studies were large ones, for instance, 1199 people trained by the project, 503 companies, 20 training providers and 22 KADIN/APINDO offices were sampled for the impact study. The instruments used were questionnaires, focussed interviews and direct observations. The results were analysed using a path analysis technique. 19. Much of the analysis in the reports does not permit relevant quantitative conclusions to be reached (see example in para. 20), which is due to the use of the path analysis technique used that expressed all conclusions as relationships between dependent and independent variables. However, some qualitative conclusions are clearly presented. The overall utility of these studies however, is disappointing. The key qualitative conclusions are: (a) the training improved the job performance of trainees but had no effect on their salaries; (b) high quality training had a high impact on improved performance of the companies but enterprises were sensitive to high training costs; (c) there was an improvement in the quality and an increase in the number of training providers; (d) the awareness of training in companies has increased; and (e) employers regarded the demand-driven approach to training as a major strength.

17 Quality of Training. This was one of the key indicators that was to be measured by the evaluation studies but because of the special analytical technique used there is no quantitative data in the report for basic criteria such as the rating of the training by the companies and trainees. There is one overall conclusion drawn from quantitative data that 'on the whole, the respondents believed the SDP training to be of reasonable quality'. However, in relation to this, problems ranked in order of importance, were identified by a minority of respondents indicate issues of: quality of training providers, quality of training instructors and their substitutes, quality of training materials, limited training time and limited training budget. 21. Cost Sharing. A key element of the design of the project was the contribution by companies to the cost of training on an upward scale of 20%, 40% and 70% per year of participation in the project. In practice, up to the 40% level was widely achieved but there was resistance to the 70% level that may have been partly caused by the economic downturn, though there was a high 70% level of achievement in East Java province. There was also resistance to the contribution by small companies as they normally qualify for full subsidization of training costs under government programs. The payments were made in cash or in-kind in the three provinces with the proportion differing between them. The large companies in the ITS program paid in cash or in-kind through the provision of facilities and instructors. The contribution to the CGTU program by the medium sized companies was mostly in cash. The contribution to the SETAS program for small companies was mainly inkind by the provision of items such as facilities and food. These funds were a potential source of 'leakage' but it is not known if this occurred as they were not audited. The general view in the operational phase is that many medium and large companies would be willing to contribute an average of 40% to 50% of the cost of training in the current recession. 22. This was the first time that such a cost sharing model was used by MOM and that they decentralized financial control to the provincial level (which however, had it problems as presented in Section C). The cost sharing model was generally acceptable to enterprises and it was considered to be effective by MOM. The Ministry has extended the use of variations of the cost sharing model to other programs together with decentralized budgeting and control. The experience gained under the project was valuable to the Ministry as it was a learning experience for them. Ministry of Manpower Component Achievement of Targets as set out in the SAR 23. Staff Training and Development Subcomponent. This program was successful and was substantially achieved. All overseas higher degree programs were completed except two doctorates that are well advanced. A number of domestic doctorate and masters degree programs started under the project will be completed after closing and MOM has provided government funds for the completion of all degrees. For those staff who have completed their higher degrees under the project, there was a good match between their qualifications and posts they held. MOM recognizes that this expenditure on higher education will considerably strengthen its capacity for policy development and management and they are continuing to fund more domestic masters programs out of government funds. The planned short-term

18 -6 - training was not implemented under this subcomponent as resources were directed to higher degree training. 24. Local Labor Market Information System (LLMIS) Subcomponent. Work on the LLMIS was delayed by the long time taken to prepare the terms of reference and the late appointment of the consulting firm. The subcomponent also suffered from shortages of counterpart funding when the economic crisis hit, which led to the cancellation of hardware for the central office and PCs and software for the district offices in the project provinces. The system in the central office is capable of serving the needs of MOM but the failure to upgrade the system as planned means that anticipated efficiencies will not be realised. For example, the district offices will continue to supply information in hard copy as in the past. 25. The achievements include: (i) networking computers in MOM; (ii) integration of the data bases in three directorates general in MOM; (iii) more ease of exchange of information and improved cooperation between the staff of the three main data bases; (iv) off-the-job and on-the-job training for relevant staff, (v) the capacity of analysis, programming and user staff to produce good quality work has been developed; (vi) establishment of an intranet facility; (vii) development of a MOM home-page on the internet; and (viii) a range of improved labor market publications. 26. Young Professionals Entrepreneur Development Program (PMP). Due to the withdrawal of the short-term training model from the project by the Government (para. 11) and the reduction in the work-plan for the medium-term model because of the late appointment of the international consultant, the scope of this subcomponent was substantially reduced. The achievements under the project were the production of resource materials for the two models and a number of relevant reports by the consultant including a review of best international practice in enterprise development. 27. As a project focused on training, the SDP has contributed to decentralization of the management of training and other labor market measures in Indonesia. After overcoming constraints and unacceptable practices (Section C), the project in the end successfully piloted planning, budgeting, implementation and monitoring of private sector training in three provinces. As a result, new roles and relationships are being established between national and provincial levels and these are being extended to other provinces for other programs. 28. After a slow start, the project satisfactorily achieved overall its targets specified in the SAR for establishing and testing a system for and implementing private sector training in three provinces. The achievement of project targets for the MOM component was satisfactory overall after taking into account the decision by the Ministry to substantially reduce the scope of the PMP subcomponent. The achievement of the Staff Training and Development subcomponent was substantial while that of the Local Labor Market Information System was satisfactory.

19 C. Major Factors Affecting the Project Ministry of Manpower Component 29. Apart from the planned loan repayment scheme not being approved for the PMP subcomponent (para. 11), at the central level, there were no major external factors affecting the project, as the other activities under this component were entirely self-contained investments in building the management and policy capacity of the Ministry. Control and responsibility for the component rested completely with the MOM and were not subject to the same unforeseen external factors that caused serious problems in the provincial component. Provincial Component Project Design and Preparation 30. The project design and preparation teams recognized the difficulty of implementing a decentralized project in three provinces where the administrative procedures involved were new to the central and provincial government officials, and the private sector staff involved. For this reason, money from a previous project (Loan 2705-IND) was used to finance teams in the three provinces to prepare their subcomponents. However, in hindsight, the administrative procedures that were prepared were not comprehensive or detailed enough to facilitate implementation. A practicable set of standard operating procedures was only used during the last eighteen months of the four year project. 31. The mechanisms and detailed procedures for the transfer of loan funds from the national to the provincial level (SPABP), release of provincial government funds, and payments for services by the Provincial Government project manager (PIMPRO) were prepared by the Borrower. Again, as these were new to the numerous central and provincial level officials involved they had to be further detailed during project implementation to work effectively. These start-up problems caused temporary but not lasting damage to the implementation of the project. Disbursements were low and training and other project activities had to be delayed during this stage of the project. Project Financing Issues 32. Problems caused by delays in the provision of project funds. Serious operational difficulties were caused especially during the first half of the project by (a) bottlenecks that impeded the flow of Bank funds to the provinces through the SPABP mechanism; and (b) the failure by some provinces to make counterpart funding available sufficiently early during the fiscal year. These delays meant that there were long time lags of about eight and ten months in the payment of PIU staff salaries during the first two years of the project and various informal measures, that did not involve Loan funds, were taken to alleviate the situation. 33. In West Java, the then PIU Director gave an interest-free loan to the staff to cover part of their salaries and operating expenses, that was repaid when the finds became available. In East Java, all staff worked without being paid until the money arrived. The problem in North

20 -8- Sumatra was handled by taking out a commercial loan to cover salaries of lower level staff and partial payment of training providers and the borrowed money had to be repaid and the interest costs serviced when project funds became available. During the first half of the project, Bank review missions concentrated on overcoming these obstacles to the flow of funds. 34. Refunds of fees by training providers. In West Java, training providers were asked to make a partial refund of fees paid to them from the Loan to the PIU to pay for services directly provided by the latter which the training provider was paid for because of the conditions of the contract. These included certification, training materials and photographic evidence to show attendance of participants at training. In the same way, the CGTUs felt justified in recouping the costs of conducting training needs analysis for their member companies and the associated travel costs from the fee paid to the training providers from the Loan. The PIU and PIMPRO siated verbally without providing documentation that the amounts repaid were about 3% of the contract amount. No evidence was obtained that this practice applied in East Java. 35. The amounts involved in these practices cannot be objectively estimated as no formal accounts of this money was kept. Private sector funding as well as Loan and Government funds should have been subject to financial control. The rebates were justified for the certification and the provision of training materials for participants, but the operating procedures should have been changed to ensure formal control of this money. The charges imposed by the CGTUs for training needs analysis and travel was clearly wrong as these items were to be covered within their budget. 36. Rebates by the PIU to the Provincial Government. It was stated to the mission, verbally and without written documentation or proof, that depending on the province, 1 to 3% of the provincial government funds provided to the project were repaid for services provided by them to the project such as official stamping of documents and photocopying. Also, verbal information without written documentation was provided that the Treasury Office (KPKN) in each province got a rebate of about 1% of the loan funds it paid out, also for services provided. No representative from KPKN was present when this information was given and they have not commented on this as it was not independently verified with them. 37. Contributions by the private sector to the cost of training. In East Java, the contribution from the private sector was placed in the control of the PIMPRO and was accounted for. In West Java and North Sumatra cash contributions from companies were paid directly to training providers, they did not flow through the PIU, though they were administratively recorded by them. 38. It seems that there was a large amount of money circulating in the project that was handled informally (written accounts were not kept) such as private sector contributions and the various refunds. The amount of money is unknown as (a) proper financial control was not kept over most of it, and (b) it has not been included in the audits. Some of the reasons for these informal practices were valid but others were not. The valid practices should have been formalized by modifying the operating procedures to include them, which was not done. There was financial control of the Loan and Government funds but these measures would

21 - 9- have ensured financial control of the private sector funds. It is not possible to give an objective view of the effect of these 'unacceptable' practices on the performance of the project as they were not disclosed until the end of the project. Award of Training Contracts 39. In September 1996 the Bank found that a small number of private training providers were winning a high proportion of contracts (concentration in the award of contracts) in all three project provinces. There were serious delays in the payment of training providers during the first three years of the project. These were mainly caused by two factors: (a) the flow of funds to the provinces was severely delayed in the first half of the project period as described in para. 32, and (b) there were serious bureaucratic problems in the Provincial Governments that were only overcome in the third year of the project. There were no serious problems with delays in payment during the last year of the project. 40. Many firms would not tender for training contracts because of this delay in being paid during the first three years of implementation. Also, large firms did not consider the budget to be high enough to make it commercially viable for them to take part in the program. As a result of these constraints, the PIUs felt strongly that they had no choice but to use those training providers that would work under those conditions. Another factor that resulted in concentration was the insistence by some large companies in the ITS program to use training providers nominated by them. 41. An unacceptable reason that caused concentration of training providers and complaints about the contracting procedures was nepotism. This is defined here to cover contracts being given to training providers in which project staff, relatives and friends had an interest. There were serious examples of this in all three provinces and as a result of letters from the Bank to MOM, different changes were made to project staff in the three provinces and bidding procedures were changed. In addition there was friction amongst the project staff and the possibility that false training proposals were prepared in North Sumatra. After a large number of training staff resigned including the Director of the PIU, the new Director subsequently refused to accept 79 training proposals from training providers on the grounds that they had been improperly prepared. Two civil suites were brought against the PIU Director and the Chairman of the PTF Board in North Sumatra by some former training staff. One suit was dismissed and the other seems to have been withdrawn. The PIUs claim (and evidence supports this), that the problems that caused the concentration in the award of training contracts were overcome during the last year of the project (the extension) as there was a greater spread of training providers used. D. Sustainability 42. Government commitment to the development of small and medium enterprises is very strong and there is a strong acceptance that human resources development is a key to growth in this subsector. For the provincial component, MOM has stated that it wishes to sustain the SDP program in a modified form in the three provinces from June 1999 and to replicate the program in other provinces in the following year. In the three provinces, employer organizations and provincial government support a continuation of the project programs in

22 -10- various modified forms. As priority is being given to social safety net programs, the SDP would possibly be facilitated in accessing the Foreign Workers Fund (fees from the licensing of foreign workers to work in Indonesia). MOM is also encouraging the SDP to access the social safety net funds through the MOM Provincial Offices and to seek funding from the provincial offices of BAPPENAS. The Ministry has decided that the proper location for these activities in the operational phase should be the Directorate of Training and Productivity rather than the Directorate of Manpower Planning and Development that was responsible for the implementation phase. 43. Financial Sustainability. A government subsidy will continue to be necessary since it is unlikely that the majority of PIUs and CGTUs can survive without this. In general, it is felt that enterprises would only be willing to pay about 40% to 50% of the total cost of training. It would be important to maintain a key feature of the program, that is, the encouragement of enterprises to participate in training and to establish a training system. A promotional activity backed strongly by the employer organisations could be beneficial to companies to prepare for and participate in the forthcoming economic upturn if it encourages them to train to meet market demand. The PIU structure is the cornerstone of this activity. Public encouragement and support for companies to train their workers is a key policy of many OECD countries. 44. The majority of the PIUs and CGTUs from the extension phase are operating on a part-time basis (while some are operating full-time) until the funding situation has been clarified. In order to improve their prospects of being viable they have reduced their staff numbers and operating costs to less than half of the costs they had during the project. They have well equipped offices. The program in North Sumatra hopes to receive financial support from the Provincial Government. 45. Institutional Sustainability. The maintenance of the capacity built up in the three provinces is dependent on government funds being made available. While there is strong commitment at present, the longer the time-lag in the provision of funds, the more likely it is that trained staff will leave the program. Due to the decentralized design of the project, there are only a small number of officials in MOM who are experienced with the program and these are not in the Directorate General that is now responsible for it. There is no capacity at present in other provinces that did not participate in the project and if it is replicated the many lessons learned from this demonstration project should be carefully applied. 46. Ministry of Manpower Component. The Ministry have expanded the Staff Training and Development program using government funds and they have a budget allocation to start twenty more officials on domestic masters programs in 1999/2000. The MOM also have a budget allocation for the LLMIS program in 1999/2000 sufficient to sustain and consolidate the successful outcomes of the project. The short-term training model of the PMP programn that was implemented outside of the project using government funds is being maintained by the Ministry. E. Bank Performance 47. During the first two years there were many administrative issues that impeded implementation of the provincial component and the Bank's supervision efforts were focused

23 on the resolution of these. Because of this, it was not until 1996 that the conflict of interest in the award of training contracts and irregularities in the contracting process were detected by the Bank and subsequently acted on by the Borrower. While the Bank asked the Borrower to take corrective action soon after discovering the conflict of interest, the Bank could have put more pressure on the Borrower to ensure that the necessary action was taken quickly enough. The changes of Task Managers adversely affected continuity of approach to project implementation. Project financing problems were not detected earlier because the provincial PIUs only began to talk about them during the final supervision mission in August 1998 and much more during the ICR mission in March Considering all of the issues that arose and given that one aim of this pilot project was to learn lessons from the experience of different approaches, overall Bank performance in project supervision was mainly satisfactory, despite some weaknesses, for this innovative project. 48. With a shift to increased responsibility in the Bank's Resident Mission, there were improvements in communication with the Borrower. This provided greater opportunity to provide much needed technical support to the Borrower in procurement, disbursement and accounting procedures that had been poorly understood by the Borrower in the earlier years of implementation and had contributed to delays in issuing letters of no objection. F. Borrower Performance 49. Borrower performance improved over time and was satisfactory in the second half of the implementation period. It was weak during the first half of implementation and in hindsight, also during preparation. As discussed in para. 31 the level of detail necessary to make the administrative procedures for the transfer of Loan funds to the provincial level and the release of provincial government funds work effectively was underestimated by the Borrower and it was only later in the project that these difficulties were overcome. A standard set of operating procedures was only used during the last eighteen months of the project (para. 30). 50. While recognising that the Provincial Governors, through the PTF Boards and PIUs were responsible for implementation of the provincial component, the national level Central PIU (CPIU) and the Project Steering Committee (PSC) were set up to facilitate overall implementation and to take responsibility for policy coordination. The CPIU and the PSC should have played a more forceful role in the facilitation and monitoring of project implementation. For example, a more timely development of the standard operating procedures (SOP) would have given provincial authorities a firm administrative framework within which to work and would have reduced the scope for conflict of interest in the award of training contracts (para. 41). Likewise, strong representation to provincial governments could have resulted in more timely provision of counterpart financing thereby, reducing the need to fund PIU operations through staff salary reductions and training provider refunds (paras. 32 to 35).

24 G. Assessment of Outcome 51. Based on the achievement of the key and quantitative indicators (Tables 5, 6 (a) and 6 (b)), the project's outcome was satisfactory for the achievement of implementation progress and development objectives for the last eighteen and twenty four months respectively, of implementation. The project was rated as unsatisfactory before this. The efforts made by the Borrower and the Bank during the first two years to overcome administrative obstacles and the adoption of standard operating procedures enabled the project to perform satisfactorily for about the final two years. The improved performance was assisted by the restructuring of the project in August Satisfactory, qualitative outcomes about the beneficial effects of private sector training and its quality were obtained from the evaluation studies but no relevant quantitative conclusions could be drawn from them due to the path analysis analytical technique used (para. 19). 52. It is encouraging that despite resistance from some employers, many enterprises were willing to join substantially with the Government in managing and investing in the training of their workers, despite poor program administration by the Government in the first half of this project. The design of the project was shown to be practicable: decentralised, private sector dominated training funds subsidised training needs analysis and training of employees. The value and importance of training in firms was promoted. The medium term implications for public training policy that the project model and incentives work are substantial. Valuable experience has been gained in the implementation of decentralized private sector training in three provinces. Many lessons have been leamed from this demonstration project. The strength of institutions (Borrower and Bank) lies in the ability to learn from experience. This has translated into better design for follow-on and other decentralized programs being implemented and planned by MOM that will be financed through a cost sharing structure derived from the one used in this project. H. Future Operations 53. The Bank's CAS continues to highlight the importance of continued emphasis on human resource development to promote economic growth, competitiveness, and structural transformation, and on a more selective, efficient use of public resources. The Bank has engaged in a comprehensive dialogue with GOI about training through the publication of the Sector Study 'Training and the Labor Market in Indonesia: Productivity Gains and Employment Growth' (Report No IND). The chapter on employer-provided training concludes 'that this mechanism is important for the creation of skills and it should be preserved and enhanced. Being demand-driven, it is highly relevant to production and is costeffective. It is the kind of training that produces the occupation-specific skills that governments have difficulty in forecasting and find expensive to provide through vocational/technical education'. The study suggests tax incentives and cost sharing between employers and employees as examples of measures that could be used to support employer training. 54. In the light of the very constrained econornic environment currently in Indonesia the Borrower and the Bank have agreed that priority for human resource development support would be given to maintaining enrollments in basic education. When the Indonesian economy

25 has improved, future potential investment in the training subsector would be guided by the 'Training and Labor Market in Indonesia' Report and the lessons learned from this project. I. Lessons Learned 55. The SAR incorporated a number of lessons learned from previous projects in the training and education sector. Several of these refer to institutional arrangements and management capacity such as: (a) provision of financial incentives to private enterprises to develop industry-led and demand-driven training programs on a cost sharing basis; (b) strengthening the planning, research and policy-making capacity of MOM to help the Ministry to develop and disseminate relevant labor market information; (c) decentralization and private sector involvement in project implementation; and (d) ensuring adequate coordination and communication among implementing agencies. The SDP built on the lessons stated in the SAR and adaptations were made during project implementation to reflect some of the lessons being learned during that stage. 56. There have been valuable lessons learned from this project and the main ones are: first, after the administration difficulties were overcome the institutional arrangements and the funding mechanisms used in the project were found to be suitable for the decentralization of the management and delivery of training at provincial level; second, many private sector employers especially medium and large ones, are willing to take responsibility for planning and organizing the training of their employees; third, though there would be some resistance especially from small firms, many enterprises accept the concept of cost sharing and it is estimated that in the present recession many medium and large companies would be willing to contribute an average of 40% to 50% of their costs of training. The experience gained by MOM under the project was valuable as under a new policy initiative, the Ministry has extended the use of variations of the cost sharing model to others of their labor market programs together with decentralised budgeting and control to provincial level; fourth, it took two years to get the special fund channelling mechanism (SPABP) used for transferring Loan funds from central to provincial level to operate smoothly as it was new to MOM and these provinces. As decentralisation continues this mechanism may have to be used more widely and therefore, in projects where it will be used care should be taken to ensure that detailed operating procedures are pilot tested during project preparation; fifth, if this project is to be replicated or related structures established that delegate management responsibility to provincial level, it would be essential that comprehensive and detailed operating procedures would be developed and approved during project preparation. These ex ante regulations would permit more ex post accountability for performance and use of funds, which should be carried out. Also effective coordination and monitoring of project preparation and implementation at central level by the Borrower are essential; sixth, the original time-scale of three years would have been right for this demonstration project to test the mechanisms used in the absence of serious administrative delays. However, based on the experience of the contributions from the private sector, providing subsidies for three years is too short a time to prepare enterprises to fully fund the cost of planning and delivering the training of their own employees, about six years would be required; and seventh, the evaluation studies were carried out at the end of implementation which allowed

26 no opportunity for corrective action to be taken about the analytical method used. A pilot test of the methodology carried out earlier in the project would have highlighted the problem with the analysis and allowed time to correct it.

27 Table 1: Summary of Assessments A. Achievement of Objectives Substantial Partial Negligible Not Appticable Macroeconomic policies Sector policies Financial objectives Institutional development Physical objectives Poverty reduction Gender issues Other social objectives Environmental objectives Public sector management Private sector development x x x x x x x x x x x Increased women and minority enrolment B. Project Sustainability Likely Unlikely Uncertain MOM Component Provincial Component x x C. Bank Performance Highly Satisfactory Satisfactory Deficient Identification Preparation assistance Appraisal Supervision x x x with some weaknesses x D. Borrower Performance Highly Satisfactory Satisfactory Deficient Preparation Implementation Covenant compliance x for last 1.5 years x x E. Assessment of Outcome Highly Satisfactory Unsatisfactory Highly Unsatisfactory Satisfactory Project Development Objectives m x from the id point

28 - 16- Table 2a Related Bank Loans/Credits Loan/Credit Title Purpose Status 1. Nonformal Education The purpose of the project was to strengthen the capacity of Closed Project (Ln. 1486) Penmas, the govermment agency responsible for out-ofschool education, to provide effective non-formal education programs by helping it overcome existing problems in administration, training, materials development, evaluation and budgeting in seven provinces comprising the majority of the population of Indonesia. In support of the above objectives, the project would finance physical facilities, regular in-service training for Penmas staff, non-formal instructional materials development, a basic learning fund, program monitoring and evaluation, and related technical assistance. In strengthening Penmas, the project would provide vocational training and basic education to about 2.5 million Indonesians in seven provinces. 2. Second Agricultural The proposed project constitutes the second phase of a Closed Training Project program initiated under Cr. 288-IND to improve the quality (Ln. 1692) and supply of middle-level agricultural manpower. The proposed project would: (a) upgrade 12 existing agricultural development schools and three existing in-service training centers; and (b) establish five new agricultural development schools and 7 new in-service training centers. The majority of these centers and schools would be located in provinces outside Java. In addition, the project would introduce courses in animal health and rural home improvement in all agricultural development schools and training centers. The project would develop and introduce an agricultural management training program and includes fellowships and technical assistance for management, training, procurement, preparation of future education projects, middle-level agricultural manpower surveys, and tracer studies. The main benefit of the project would be to help ease the shortage of qualified middle-level agricultural manpower. 3. Public Works The main objective of the Public Works Manpower Closed Manpower Development Project is to improve the effectiveness of the Development Project Ministry of Public Works (MPW) construction activities (Ln IND) through manpower training. The project aims to: (a) strengthen the management of MPW staff training and related planning activities; (b) establish a Curricula and Materials Development Unit in the MPW; (c) expand and upgrade the MPW's training facilities and staff training program; and (d) provide experimental training for small- and medium-sized contractors and mandors (labor subcontractors who directly hire and supervise construction labor). A second objective is to strengthen Government planning and management capabilities through provision of fellowships for key staff of economic and policy-making agencies. 4. Second Non-Formal The project objectives would be to increase access to Closed Education Project nonformal education, improve the quality of nonformal (Ln IND) education programs, and strengthen the management capability of the Ministry of Education's Directorate of Community Education (Dikmas). To achieve these

29 - 17- objectives, the project would& (a) expand learning programs in basic education and employment-oriented training; (b) improve the quality of educational programs by developing relevant learning materials and providing staff training. Furthermore, it would; (c) improve program implementation by introducing better planning, monitoring and evaluation systems and an applied research program at Dikmas. The project would benefit about 800,000 Indonesian illiterates and school dropouts in 17 provinces by providing basic education and employment-oriented skill training. 5. Third Non-Formal The project would contribute to improving the human Education Project (Ln capital of the poor, through enhancing their basic education IND) and income-generating skills. The project would have three parts: the Basic Program (BP) would continue the provision of basic literacy to illiterates and primary school dropouts; the intensive program (IP) would implement a special strategy especially targeted to low-income beneficiaries; and the national support services component would strengthen the Directorate of Community Education (DIKMAS) in their capacity to deliver the IP. The IP components include: upgrading of private vocational courses, including scholarships; expansion of an informal apprenticeshiprogram; training in small business management skills and technical assistance to "Kejar Usaha" (KU) income-generatingroups. Closed 6. Institutional Development & The Public Works Institutional Development and Training Closed Training Project Project hopes to improve the coordination of planning and (Loan 3112-IND) the implementation of public works activities. The project, which will build on investments in training programs, systems and procedures, will comprise three components. The first will provide (a) Staff and institutional development; improved management and coordination of training; strengthening of provincial public works offices; and public works staff training. The second component will include: (b) Improved Management Information Systems (MIS); refinement of selected subsystems of the MIS; a system for uniform reporting of planning, programming and control of public works programs; and improvement of procurement procedures, financial control, and audit, library and public information services. Lastly, the third component will provide: (c) improved quality control for constructing and equipping laboratories, as well as staff training through the provision of testing procedures and facilities.

30 Table 3: Project Timetable Steps in Project Cycle Date Planned Date Actual Identification June 3, 1991 April 15, 1992 Preappraisal January 15, 1992 August 19, 1993 Appraisal April 5, 1993 October 28, 1993 Negotiations October 11, 1993 January 31, 1994 Board presentation December 7, 1993 March 25, 1994 Signing April 26, 1994 Effectiveness June 13, 1994 Project Completion December 31, 1998 Loan Closing December December 31, 1998 Table 4: Loan/Credit Disbursement: Cumulative Estimate and Actual (US$ million) FY94 FY95 FY96 FY97 FY98 FY99 Appraisal Estimate Actual Actual as % of adjusted estimate Date of final disbursement May 5, 1999

31 19 - Table 5: Key Indicators for Project Implementation Indicators Expected Improvements through Duration of Proect SAR Actual PIU/PTF Board Management 1. Cycle-time for training The PIUs/PTF Boards will Processing time for award of proposals from industry reduce the cycle time for contracts was satisfactory. processing of contracts by 10% each year of the project 2. Cost-recovery within The PIUs/PTF Boards will 80% and 60% targets project training activities reduce the Government achieved in three provinces. financing share from 80% the 30% partially achieved in first year, to 60% the second one province. Satisfactory year, to 30% the third year of overall. the project. 3. Board efficiency The PTF Boards will meet as Not achieved. They met as prescribed by agreed required but were not well guidelines and there will be attended. at least 80% attendance at all meetings. 4. Program The PIUs/PTF Boards will This was not systematically promotion/awareness systematically increase the measured but promotional activities. number of activities were actively promotion/awareness increased throughout activities by 15%per year implementation. over the duration of the project. 5. Employer satisfaction The PTF boards will conduct Evaluation studies indicated with training and advisory an annual survey of client overall employer satisfaction services. companies and will receive a - satisfactory. high score that indicates a high level of employer satisfaction. 6. Capacity to maintain The PIUsIPT Boards will Comprehensive actual costs internal costs acceptable for provide records showing recorded in East Java. service delivery. actual costs of all services Actual costs using Loan and provided. Provincial funds and notional costs using employer funds maintained in West Java and North Sumatra - satisfactory overall. Company Group Training Services 7. Increase in level of CGTU Existing CGTUs will Company membership company membership increase their company targets substantially achieved membership by 15% per year during the original phase but through duration of project number of CGTUs deliberately reduced in the extension to assist viability -

32 -20 - satisfactory. 8. Increase in availability of The CGTUs will increase The training targets in training services offered number of assisted training companies were satisfactorily through CGTUs activities within companies achieved by 25% each year of the project Job-Seeker Training 9. Participation of private The percentage of Job Seeker All JST training was sector service providers Training services provided provided by private providers through private organisations will increase by 15% each year -satisfactory. Young Professionals Entrepreneur Development Program (PMP) 10. Access of participants to The PMP will provide credit MOM tried but failed to get credit to 100% of all graduates who approval for a credit apply to finance new enterprise start-up guarantee scheme and the Loan Agreement was amended to delete this requirement from the project. Local Labor Market Information System (LLMIS) 11. Information A range of dissemination Satisfactorily achieved. dissemination and quality methods to be used, such as reports, newsletters and fliers. The quality of information will improve 12. Satisfaction of users with MOM will conduct two User satisfaction not LLMIS surveys of users during the formally measured. Informal project. User satisfaction feedback was good. will increase Staff Development 13. Return on investment in Following completion of Achieved. staff training training, 90% of all fellowship graduates will receive key positions within the MOM

33 -21 - Table 6(a): Quantitative Indicators for the Operation of the Provincial Component Indicator Unit of Measurement SAR Actual North Sumatra Province Establishment of CGTUs No. of CGTUs (87%) Appointment of new training managers Persons 100 NA Training provided through CGTUs Persons 14,340 11,945 (83%) Training provided through non-cgtus Persons 9,775 9,440 (97%) Training provided for job-seekers Persons (100%) Training-related placement % 85% 97% Advisory services for small enterprises No. of firms (28%) Training for employees of small enterprises Persons (100%) Overseas short-term training Persons (100%) West Java Province Establishment of CGTUs No. of CGTUs (93%) Appointment of new training managers Persons 300 NA Training provided through CGTUs Persons 16,865 18,705 (111%) Training provided through non-cgtus Persons 10,564 8,023 (76%) Training provided for job-seekers Persons (100%) Training-related placement % 85% 99% Advisory services for small enterprises No. of firms 1,440 1,610 (112%) Training for employees of small enterprises Persons 2,160 2,412 (112%) Overseas short-term training Persons (100%) East Java Province Establishment of CGTUs No. of CGTUs (93%) Appointment of new training managers Persons 300 NA Training provided through CGTUs Persons 24,610 15,962 (65%) Training provided through non-cgtus Persons Training provided for job-seekers Persons (99%) Training-related placement % 85% 99% Advisory services for small enterprises No. of firms 1,908 1,908 (100%) Training for employees of small enterprises Persons 3,240 3,150 (97%) Overseas short-term training Persons (100%)

34 . 22- Table 6 (b); Quantitative Indicators for the Operation of the Ministry of Manpower Component Indicator Unit of SAR Actual Measurement Young Professionals Entrepreneur Development Program (each of 5 participating provinces) Graduates from short-term model (STM) No. of graduates 900 Not applicable Businesses started among STM graduates No. of businesses 630 Not applicable Jobs created through the businesses started No. of jobs 1,418 Not applicable Graduates from medium term models (MTM) No. of graduates 200 Not applicable Businesses started among MTM graduates No. of businesses 140 Not applicable Jobs created through the businesses started No. of jobs 420 Not applicable Staff Training and Development Persons started overseas Ph,D. fellowships Person 3 3 (100%) Persons started overseas M. A. fellowships Person (88%) Persons started domestic Ph. D. programs Person 3 2 (67%) Persons started domestic M.A. programs person (100%) Domestic short-term training person-month Local Labor Market Information System Annual establishment surveys survey 4 Annual labor force surveys survey 2 Semi-annualabor market information reports report 6 In-country sort-term training person-week Overseas short-term training person-week

35 -23 - Table 7: Studies Included in the Project Study Purpose as Defined at SAR Status Appraisal Ministry of Manpower Component Industry Identification Study Support for PMP medium- 25 studies Model not term training model implemented by GOI Establishment Survey Local Labor Market 4 surveys Completed in three Information System (LLMIS) provinces (3 surveys) improvement Labor Force Survey LLMIS improvement 2 surveys Used government surveys as source of data by agreement with Bank Research on Labor Market LLMIS improvement 3 studies Conducted informally Information Establishment Survey Preparation of PTF in South 2 surveys Activity cancelled Sulawesi from the Loan Agreement Evaluation Studies Impact evaluation of the 2 studies 2 studies: (a) Record project of Experiences, and (b) Impact Evaluation, by agreement with Bank Technical Audits Auditing of project 12 audits Partially complied performance with. Some audits not satisfactory.

36 -24 - Table 8 (a): Project Costs Appraisal Estimate (US$M) Actual/Latest Estimate (US$M) Item Local Foreign Total Local Costs Foreign Total Costs Costs Costs A. Provincial Component Company Group Training Services Individual Company Training Services Targeted Training of Job Seekers Small Enterprise Training and Advisory Services Project Administration Subtotal Provincial Component B. Ministry of Manpower Component Young Professional Entrepreneur Development Program Local Labor Market Information System Staff Training and Development Preparation of PTF in South Sulawesi Evaluation Studies Project Administration Subtotal Ministry of Manpower Component Total Baseline Cost Physical Contingencies Price Contingencies Total Project Cost

37 Table 8 (b): Project Financing Appraisal Estimate (US$M) Actual/Latest (US$M) Estimate Source Local Foreign Total Local Foreign Total Costs Costs Costs Costs IBRD GOI and Private Companies Total

38 TABLE 9: STATUS OF LEGAL COVENANTS Agree- Section Covenant Present Original Revised Description of Covenant Comments ment Type status fulfillment fulfillment date date Boardl C 08/31/ Provide action plan for PMP. Boardl C 08/31/ Establish private sector management committees and provincial coordination units for PMP. Boardl C 10/10/ Provide TORs for evaluation study. Boardl C 2/15/ Carry out evaluation study. Boardl CP 10/10/ Carry out technical audits Technical every six months and audit provide copies to the completed. Bank. Not satisfactory. Boardl C - Training funds to implement Part A of project in accordance with satisfactory operating procedures and eligibility criteria. Covenant Class: Status: I = Accounts/audits 8 = Indigenous people C = covenant complied with 2 = Financial performance/revenue 9 = Monitoring, review, and reporting CD = complied with after delay generation from beneficiaries 10 = Project implementationot CP = complied with partially 3 = Flow and utilization of project covered by categories 1-9 funds It = Sectotal or cross-sectoral 4 = Counterpart funding budgetary or other resources 5 = Management aspects of the allocation project or executing agency 12 = Sectoral or cross-sectoral policy/ 6 = Environmental covenants regulatory/institutional action 7 = Involuntary resettlement 13 = Other

39 -27 - TABLE 10: COMPLIANCE WITH OPERATIONAL MANUAL STATEMENTS There was no significant lack of compliance with an applicable Bank Operational Manual Statement (OD or OP/BP) TABLE 11: BANK RESOURCES: STAFF INPUTS FY Tota Preappraisal Appraisal Negotiations Supervision Completion Total

40 -28 - TABLE 12: BANK RESOURCES: MISSIONS Performance rating Specialized staff Imple- Devel- Stage of project cycle Month/ Number of Days skills mentation Opment Type of year persons in field represented La status lb Objectives Problems /c Preidentification Identification/Preparation 4/ Economist; Training Spec. Educator Preappraisal 8/ Economist; Training Spec. Project Assistant Appraisal 10/93 Negotiation Postappraisal 2/94 Postappraisal Board approval 3/25/94 Signing 4/26/94 Effective 6/13/94 Supervision 08/ Training Spec. 2 2 Implementation Spec. 11/ Economist 2 3 Implementation Economist delays Operations Spec. Consultant 02/ Training Spec. 3 3 Implementation Consultant delays 10/ Economist 3 3 Implementation Vocational delays Training Spec. Operations Spec. 01/ Economist 3 3 Unavailability Vocational of funds Training Spec. Operations Spec. 8/ Economist, 3 3 Implementation Training Spec. delays Operations Asst. 8/ Training Spec. 2 2 Economist Operations Spec. 08/ Economist Operations Spec. 2 2 Completion 3/ Consultant 2 2 la DO: Disbursement Officer; EC: Economist; EN: Engineer; FA: Financial Analyst; LC: Legal Counsel; 00: Operation Officer; RA: Research Analyst; RS: Resettlement Specialist; TE: Transport Economist; TS: Transport Specialist. /b 1: Highly satisfactory; 2: Satisfactory; 3: Unsatisfactory. /c Typical problems included: implementation delays in technical assistance and studies.

41 -29 - ANNEX A Contribution of the Ministry of Manpower of the Republic of Indonesia to IMPLEMENTATION COMPLETION REPORT SKILLS DEVELOPMENT PROJECT (LOAN 3721-IND) The project became effective on June 10, Originally the cost of the proposed project was estimated at US$ 27.7 million. However the delays in the start-up the implementation had made the planned target reduced, which in turn caused the reduction in planned expenditure. By June 1997, at which the original plan that the project would have been closed, the government cancelled as much as US$ million. Thus by the time when the extension of the project was approved, the total budget estimated was at the position of US$ Again, during the one-year extension period, the project saw that a further reduction in expenditure could be made. A further cancellation of US$ was concluded during the period of the extension. Thus, from the original estimation of the loan budget of US$ 27.7 million, the project completed with the total of US$ million. Thus, the project had approximately utilised 56 percent of the original calculation. At the time in the beginning of the project, there were two parallel sets of activities underway. First, putting in place the necessary administrative measures to enable the provincial training funds to operate and the preparation of detailed sub-component budgets. Second, implementation of a program of key preproject activities to prepare for full-scale implementation, funded from the extension of the Manpower Development and Training Project (Loan 2705-IND). The original agreement of SDP was that it was to be implemented over a three-year period, from 1994 to Nevertheless, due to the difficulties encountered during the start-up, the project was extended for a year until 31 December 1998 to provide ample time to complete the targets and to gather enough experiences to test the system. 1. Project Objectives and Components The project had three key objectives: 1. To develop and implement innovative skills training schemes within private enterprises in three provinces (West Java, East Java and North Sumatra) for possible replication in other parts of the country in the future, as part of an overall training program to promote productivity and efficiency of the economy. 2. To support a special training program for unemployed university graduates in five provinces (the three mentioned above plus South Sulawesi and Bali) 3. to strengthen policy making, research, planning and management capabilities of the Ministry of Manpower (MOM). The objectives were not fully achieved particularly for the second objective, there was no training implemented.

42 -30- The SDP operated at two levels: national and provincial. The national or central component consisted of the following major sub-components: 1. Staff Training and Development 2. Young Professional Entrepreneurship Development Program 3. Local Labour Market Information System 4. Start up of replication of training fund scheme in South Sulawesi (cancelled) The Provincial Component, which was the major thrust of the SDP set up its objectives, as the following. * start-up new training programs and activities in medium firms by establishing company group training units (CGTUs) * start-up new training programs and activities in individual companies by supporting them to employ new training managers, and provide limited staff and expenses to support KADIN/APINDO to promote industry-based training (Individual Training Service/ITS) - support targeted programs to train job-seekers for identified vacancies with specified placement targets in jobs related to the training (Job Seeker Training/JST) * establish pilot programs to provide training and consultant services to small-scale industries (SETAS). 2. SDP Organisation The project steering committee had overseen the project through a central project implementation unit (CPIU) which served as the overall secretariat to the national and provincial component's base at the Ministry of Manpower (MOM). Staffng in the CPIU was prepared by the managementhrough a decree issued by the Project Director of SDP who is also the Chairman of the Board of Manpower Planning and Development. At the provincial level, the project was supervised by a provincial training fund board (PTFB) in each of the three pilot provinces: West Java, East Java and North Sumatra. PTFB was established by the Governor's Decree in each respective region. The PTFB implemented its daily operation of the project through a project implementation unit (PIU). The PTFB had a joint and equal representation from the government and private sector, and was usually chaired by the head of the provincial chamber of commerce and industry (KAD1NDA). The PIU was headed by a private sector Director from the employer's association in the province. The staff of the PIU were generally full time private sector recruits. In addition to the training officers and specialists of the PIU, there were training managers and assistant training managers recruited and trained to develop the CGTUs. For the SETAS component, training advisers and consultants were utilised. Since government funds were involved, the project manager (pimpinan proyeklpimpro) and treasurer (Bendahara Proyek) were required in the project structure. Even though the project was effective in June 1994, the full-scale implementation only started in October Thus, the start-up of the project had started very late.

43 Inter-relationship with Other Agencies Due to the nature of the project, a wide scope of relationship was established, ranging from other government ministries/agencies, KADIN(DA) to private sector. For one who hardly knew about financial procedure particularly applying the SPABP mechanism, it seems very complicated and frustrating. SPABP mechanism requires the involvement of different ministries/agencies. The original understanding was, the role of CPIU was only a liaison between the PIUs and the Project Director. It would not co-ordinate the provincial activities. The expectation from the regions towards the CPIU, however, had been higher than this. We were expected to have a bigger role and co-ordinating role while the staffs were inadequate in particularly in qualitative terms. 3. Project Implementation and Achievements Central Component Even though the loan agreement was signed in June 1994, the starting date of the implementation of central sub-components varied. The implementation of central components has suffered much delays. Except for staff training and development, all central components began in much later date. The following was the central component's implementation process. a. Staff Training and Development * This sub-component aimed to increase the capability of MOM staffs through the provision of fellowship for master and PhD degrees, both in the country and overseas. * As pointed out in the Staff Appraisal Report (SAR) the implementation of this activity was to be carried out by a contractor. Thus, the provision of fellowship was assigned to a consulting firmn through a procedure according to the GOI and the World Bank regulations. * The process of bidding for many of us, CPIU, was still new. In the beginning we were still very perplexed to what should be done. But, under the supervision of the management (Board of Manpower Planning and Development), the course was going smoothly. * The consultant firm winning the project was: PT Salatindo Graha Kreasi (PT SGK). The appointment of the consulting firm, at least, reduced the burden of the CPIU. This activity was the one, which started with almost no delay like the other central components. The assignment was to provide fellowship to MOM employees for Master/PhD degrees both overseas and domestic. * A number of candidates had already begun their studies, both overseas and in the country, while the project was still in its preparation stage. There was an urgency to speed up the start-up of the implementation of this component in order to maintain the time frame of the project implementation and the study duration. * The relevant units in the MOM implemented this activity in a co-ordinating manner. They were the Board of Planning and Manpower Development (as the unit managing SDP), the Bureau of Personnel

44 -32 - (under the Secretary General of the MOM, as the implementing unit for personnel management), and the Centre of Personnel Education and Training of the MOM (as the project manager and treasurer for this Staff Training and Development activity came from this office). * The procedure of preparation for the students, such as registration for candidates was done by the Bureau of Personnel together with the CPIU. The criteria of the awarding the fellowship to candidates were relatively tight, covering academic test (locally called Tes Potensi Akademik/TPA). The TPA index gives indicator of the level of one's academic potential. * Other criteria were English test (TOEFL), age limit, and nomination from the applicant's working unit. The difficulties in finding suitable candidates were encountered. The enthusiasm of candidate had been so great, yet they were not allowed to register their study interest to CPIU/Bureau of Personnel directly. They were supposed to be nominated by their superior (unit of work). Many, however, tried to contact us directly and individually. At that time the standard procedure was not in place yet. The registration and selection were done by the relevant unit, the Bureau of Personnel, which then established a selection team that were represented by different working units. * The firm together with the CPIU was to find universities. Even though the Standard Operating Procedure (SOP) was not in place, yet after going through trials, the procedure seems now fell in place. The achievement of the sub-component is as follows. Table 1. The achievement of the Staff Training and Development, SDP. Types of Target Real isation Note Scholarship l Domestic Master Degree * 35 employees have graduated, the rest are still _studying Doctorate 3 3 * One has already graduated and resumed her position in the MOM * One withdrew from the programn due to personal reason. Overseas Master Degree * A change was made to convert two master degree scholarships for doctorate scholarship. * All the employees of this programn had completed and resumed their posts. Doctorate 3 3 * One has completed and two are still studying. b. Young Professional Entrepreneurship Development Program (YPEDP).

45 * This activity was actually inspired by the aspiration of the MOM to promote the Young Professional Development Entrepreneurship Development Program in its effort to minimise the unemployment rate particularly among the university graduates. * The idea was supported by the success achieved in the implementation of the respective pilot project. The said pilot project was funded by the International Labour Organisation for the Directorate of Self Employment and Job Expansion in the Directorate General for Manpower Placement Development of the MOM through its activity INS/94/003 and further extended to INS/94/004. The pilot project covered Bali, West Nusa Tenggara and South Sulawesi. * A pre-project activities program was funded by the ILO and the outputs include: (a) papers used by MOM for provincial briefings which were carried out in all five PMP project provinces; (b) draft project management and operations handbooks, and a program promotional kit; (c) selection of the universities for the first year in the five provinces; and (d) training of staff from MOM and the selected universities. The Directorate of Self Employment and Job Expansion in MOM, which was responsible for PMP, had prepared standard operating procedures for the program which was also being implemented outside of the SDP. * The condition of the Loan Agreement of SDP for the YPEDP was the MOM had to prepare the Guaranteed Loan Repayment Scheme, which would have used ASTEK fund (the acronym of Asuransi Tenaga Kerja/Labour Insurance). Originally the TOR required the establishment of a credit scheme before the activity began. Nevertheless, the negotiation with the other related ministries at the time resulted that it was difficult to use the ASTEK fund. According to the Ministry of Finance the ASTEK fund could not be used. Thus, there was some stagnancy of the activity for while. * Another uncertainty arose created by a new decree (No. 316/1994) of the Ministry of Finance (MOF) about the use of 1-5% of the profits of all state-owned companies (BUMN). The preparation had stopped to wait for its implications. At that time, the World Bank mission endorsed the decision that MOM not to start the PMP until US$0.5 million was available to support the guarantee scheme. * The correspondence with the task manager of the SDP did not yield solution, as the World Bank was still adamant about the idea that credit scheme should first exist. The GOI at last after discussion between the relevant unit with the related ministries, came up with the new proposal to use this new credit which already exists from the state owned companies (BUN) to provide 1-5% of their net profit for small scale industries. * In accordance with this decree from MOF the credit was to be used for established small companies. If this was the case, what about the new start-up businesses which the YPEDP would produce? The MOM tried to negotiate for some optional arrangement. These included the provision of small loans either from ASTEK or the provincial development banks (Bank Pembangunan Daerah/BPDs) in order to bridge the gap of being a start-up business to an established business. Thus by the time the business unit was deemed eligible to be classed as an established business allowed them to fall within the terms of the MOF decree. * At the time of the appraisal, the Young Professional Entrepreneur Program was to be implemented only in five provinces. The MOM informed the mission that the Government was now implementing the program nation-wide, with the funding for the other 22 provinces coming from its own resources.

46 - 34- * The ILO had earlier been selected as the consultant for TA under this component, although a dratt contract was still unsigned. The MOM wished to review the type and extent of the TA required in light of the Government's decision. The mission indicated it would like to receive information on the outcome of this review as soon as possible given the limited time left to implement the project. * As part of this review, the MOM has already raised two issues with the mission: a) the location of a proposed technical assistance unit, and b) the retention of original receipts of expenditures if the ILO was to be the contractor. The MOM considered that any technical assistance unit should be located in Jakarta and that the goverrunent should retain original receipts of expenditures as required by government regulations. The mission endorsed the MOM's view in both matters and conveyed this directly to ILO representatives. * The ILO had already prepared the Terms of Reference together with the technical unit. From the start, as stated in its correspondences, the ILO was intended to be the contractor for this project. The constant changes in the TOR have resulted in lengthy process. * By the time we agreed with this proposal, then another problem came up. As original plan this component was to be contracted to the ILO. The appointment of ILO was mainly because the MOM had a pilot project for this kind in a number of provinces, which was funded by the ILO. The plan also to utilise the consultant that had been working with the MOM before with the ILO. However, during our negotiation with the ILO for the contract, there was one particular clause that was not acceptable both by GOI and ILO. It was stated that" the advance payment of 50% should be paid to ILO prior to the project". While the GOI regulation (Presidential Decree/KEPPRES) regulates that only 20% should be paid for the initial stage. The ILO after consulting the ILO headquarters in Geneva also could not accept to reduce initial payment, they remained adamant towards this figure. So, we both agreed that we could not go on with this negotiation and we terminated the process. * The GOI intended to hire an international consultant on individual basis, yet there had been no progress for this component. * The mission recommended that the government either cancelled or modified the nature of the Young Professional Entrepreneur component. At the time of the SAR, the program was not being carried out nation-wide. Thus, pilot testing a long and short version of the program as proposed in the MOM component of the project was potentially useful. * The consultant of the ILO (Mr. Lammottee) who was supposed to work with us under the ILO was still available to be hired on individual basis. We tried to contact him for a couple of months. He was working somewhere that it was difficult to communicate. By the time he was available, we the GOI finished our evaluation to assess his billing rate. Based on his CV the salary rate we could offer was under his expectation (US$ 3,500). He turned down this offer. For several months we looked for a replacement which was rather hard. The CPIU exercised some efforts including contacts some other consultants, which then be followed by faxes but there were no responses; contacted another project, in the hope they had another international contact. The last resort was, we asked through the World Bank, requesting their assistance. Eventually we obtained a number of candidates from the DEETYA of Australia. The Team of Evaluation then reviewed the CVs. 1 I Team Evaluation is established for the purpose of evaluation of consultant(s) to be recruited in the project. The team is appointed by the related project manager, usually consists of representatives of Bureau of Planning (of the Secretary General), of the related project, of the CPIU, and other related unit.

47 * Based on the evaluation, the decision reached was to appoint Mr. Paul George Di-Masi to be the consultant for YPEDP. The negotiation then began from early August 1997 and eventually he was officially recruited in November 1997 for 11 months. Achievement The international consultant had improved the modules for promotion, selection and follow up of the program. The respective technical unit conducted a national seminar to promote and disseminate these outputs. c. Local Labour Market Information System (LLMIS) The process of making the Terms of Reference for the LLMIS final had taken so long time. The technical unit (Centre of Information) had a series of discussion with the SDP task manager during their supervision visits together with the former consultant for SIPPTEK 2 project. A number of correspondences occurred to complete the expected TOR, which was finally approved by the World Bank. The following step was then to establish a Team of Evaluation. The following were the steps to deal with the preparation of this activity. * The project started with the announcement for bidding in the newspapers of Suara Pembaharuan and Jakarta Post of 24 May * The CPIU established the long list from the firms interested in the project. There were 12 companies registered. We finally came up with a short list of six consulting firms. This short list was sent to the World Bank for approval. * We distributed the invitation, TOR as well as Plan and Work Condition to the short listed firms to allow them to make proposal both technical and budget proposals to be reviewed. The time given was two (2) weeks. The team of evaluation worked on this assessment to make evaluation based on a number of criteria. The technical criteria consisted of consultant hired, substance, understanding of the project et cetera. Apart from that, a budget proposal was assessed. * The selection of the winner of the bidding was PT Multi Area Conindo (PT MACON). The decision of the bidding was issued by the letter of the Director of the Project of 31 October * Even though the consulting firm has been appointed, a new problem came up. The international consultant for this activity who was supposed to be the team leader for this activity would not be available. This also happened with his partner, another international consultant. The reason was, there was too much delays and they found it difficult to always re-schedule their plan. Thus, there was a vacant time while waiting to get replacement. Efforts to find their replacement began. After the firm found a replacement, we have to ask the approval from the World Bank. 2 This was under the Manpower Training and Development Project (World Bank Loan 2705-IND).

48 -36- * During the preparation process, after a couple of months, the team leader and the relevant unit thought a new activity was deemed to be important for LLMIS, this was: Geographic Information System. * We sent a letter asking the approval from the World Bank for another international consultant ofgis. Once the NOL issued, the consulting firm got really accelerated and the activity then went on its track. Achievement: The first stage of the activity had achieved the planned target, this included equipment procurement, various training, publications, and network system. In the middle of the implementation, the project had been able to introduce the Geographic Information System (GIS). In the second stage of the LLMIS (the extension period) the development of internetlintranet was carried out, more publication, training, and the GIS was further introduced in the three provincial offices. The project saw the significant progress through the consolidation of the Ministry of Manpower's database. The LLMIS demonstrated a good impact on the technical unit. Provincial Components In the pilot provinces, West Java, East Java and North Sumatra, the activities of the project have started to pick up only around the beginning of The performance of the provinces approximated full accomplishments for all the components, even exceeding the adjusted target in some cases. While the average output per year in each of the provinces were affected by the start-up difficulties, the normal capacity was probably closer to the rate of output in the extension year. The SDP has completed 2360 courses for 72,130 trainees from July 1994 to August The training activities started to accelerate only by the last two (2) years of the project as shown in Table 1. This improvement was mainly due to the number of reasons: * easier access to the training funds due to more streamlined operations * more sufficient operating budget * better capabilities of training managers * better co-ordination between the project and the regional government and * improved level of operations and supervision within the PIUs. Table 2. Training By Pilot Provinces and Component, July 1994-August 1998 West Java 18,705 8,023 2, East Java 15, ,

49 -37- North Sumatra 11,945 9, ,533 Total 46,612 18,183 6, , The Bank's Performance Bank performance in project preparation and supervision was generally satisfactory. The changes of task managers during the implementation however had brought inconsistency in its supervision particularly for the regional components. It was also a problem in the case when we expected an immediate response from the World Bank that sometimes took a long time to issue the no objection letter (NOL). Constructive working relationship was maintained with the borrower, and during the implementation sufficient to be able to resolve the problems arose and put the objectives on track. 5. Lesson Learned The project design of using the SPABP mechanism in fact had caused prolonged delays in the start up of the project. In addition to that the cost sharing model which required the contribution of the central government (loan), the regional government and the private sector had taken a longer time to put the fund in place on time. The delays of fund issuance had made the cost-sharing model through the SPABP mechanism so complicated. The unavailability of the Standard Operating Procedure in the beginning of the project had also cause misinterpretation of the procedure in the field. 6. Project Sustainability Government commitment to sustain the central components is high. The Staff Training and Development is now being continued. The provision of scholarship for domestic study is continuing using the government budget. The provision of fund for Local Labor Market Infornation System (LLMIS) exists particularly to continue the publication and system development. The implementation of the Young Professional Enterpreneurship Development Program continues on throughout 27 provinces. Regarding the regional components, each former SDP has its own approach to sustain the activity. The West Java SDP established the TRAFFIS to continuously increase the awareness of employers on training issue. In this post-sdp, it is most likely that TRAFIS is to provide its services as training provider. In East Java, the post-piu of the East Java SDP acts as training provider. Some formercgtus has been involved in training program with the East Java Regional Office of Manpower. In North Sumatra, the regional government most likely provides fund to continue training program for SDP. 7. Recommendation To start any new activity a preparation stage should be provided to the staffs. The selection of staff has to be considered. Training such as filing management, letter writing, correspondence, computer should be taken into consideration.

50 -38 - It is deemed necessary to have a set of communication equipment and facilities. This is particularly true considering the nature of the project, which covered not only the central components, but also the regional components where intensive communication was very crucial to facilitate the implementation. Misunderstanding and misinterpretation were likely to occur when there was no Standard Operating Procedure (SOP). The SOP should have been prepared during the preparation stage and should have been continuously reviewed during the implementation stage. The fact was the SOP and the Criteria for Success for regional components had been finalised after more than two years after the project started. This circumstance had led to some problems. The planner should bear in mind that preparation stage should be well planned. If the cost-sharing model for training program is to be adopted by the government, a new approach should be developed to avoid delays in fund reimbursement. In this way, the original scheme of cost sharing would be efficiently developed and applied. This would certainly give positive impacts, not only to industry (employers), but also to training providers to be more competitive.

51 -39 - ANNEX B NONESIA SKILLS DEVELOPMENT PROJECT (LOAN 3721-IND) IMPLEMENTATION COMPLETION REPORT MISSION AIDE-MEMOIRE 1. A mission by Eamonn Darcy (consultant) visited Indonesia during March 14 to 26, 1999 for the preparation of the Implementation Completion Report (ICR) for the Skills Development Project (SDP). Discussions were held with the Ministry of Manpower (MOM), BAPPENAS, and relevant authorities in the three project provinces (East Java, West Java and North Sumatra). This aide-memoire reflects the Terms of Reference issued by the Bank and other issues that arose during the mission. The mission would like to express its appreciation for the cooperation and kind hospitality extended to it in Jakarta and the project provinces. The mission's findings and conclusions are summarised below in draft form. Implementation Completion Report 2. One purpose of the mission was to provide advice to the MOM to prepare its own evaluation input to the ICR. A final report for the provincial component of the project was received from MOM on March 12, A summary report not to exceed ten pages is to be prepared by the Borrower and submitted to the Bank by the end of April This summary will be included, unedited, in the ICR. It would be an independent evaluation by the Borrower of: (a) the degree of achievement of the project's objectives, (b) prospects for the projects sustainability, (c) Bank and Borrower performance, (d) project outcome, and (e) the plan for the project's future operation. A number of key issues highlighted in this aide-memoire would also require careful attention in the ICR. 3. The mission is of the view that the summary report can be prepared from the final reports of the provincial component and of the central subcomponents in MOM. It suggests that this report would be best written by somebody from the PCIU who was closely involved in the implementation of the project. The mission urges the Borrower to take this opportunity to ensure that its views are incorporated into the ICR. 4. Another purpose of the mission was for the Borrower to assist the Bank to prepare the ICR and to provide the necessary information to the Bank to complete various tables in the ICR. The assistance provided by the Borrower to the Bank was satisfactory. 5. The schedule (to be agreed) for the preparation of the ICR is as follows: * the MOM will provide its summary evaluation report to the Bank by the end of April 1999; * the Bank will prepare a draft of its ICR and submit it to the MOM for comment by May 15, 1999; and * the MOM will provide comments on the Bank's draft ICR by May 31, Study to Record the Experiences and to Evaluate the Results of the SDP 6. Two inter-related studies were carried out by local consultants for the provincial component: (a) Preparation of a record of experience for the SDP, that consisted of two parts:

52 -40- * Report A: Comprehensive Record of Experiences; * Report B: Summary of Records of Experiences and Documentation of Major Issues; and (b) The Impact of SDP that is called Report C. An international consultant prepared a final report for the GOI using the three reports from the two studies. This final report summarised the findings, identified issues, presented policy options, and an action plan for sustainability of the project. This final report was informed by the conclusions of a seminar held to discuss a draft version of the report and Reports B and C, in December The two studies were large ones, for instance, 1199 people trained by the project, 503 companies, 20 training providers and 22 Kadin/Apindo offices were sampled for the impact study (Report C). The instruments used were questionnaires, focussed interviews and direct observations. The results were analysed using a path analysis technique. 8. Much of the analysis in Reports A, B and C does not directly inform the reader of the quantitative findings of the surveys, this seems to be due to the use of the path analysis technique. However, some qualitative conclusions are clearly presented. The overall outcome of these studies that were intended to evaluate the project is therefore, a major disappointment. The issues, recommendations, policy and sustainability options presented in the final report have been weakened by the unclear evaluation findings. Quality of Training 9. This was one of the key indicators that was to be measured by the evaluation studies but the findings suffer from the weakness described in para. 8. There are no figures in the report for the rating of the training by the companies and trainees. There is one overall conclusion based on quantitative data (available but not presented in the report) that 'on the whole, the respondents believed the SDP training to be of reasonable quality'. However, problems ranked in order of importance, were identified by a minority of respondents that were around issues of: quality of training providers, quality of training instructors and their substitutes, quality of training materials, limited training time and limited training budget. Turnover of Staff 10. The turnover of staff was high in the West Java and North Sumatra, for instance, in West Java, there were four different training managers for one post, and five different training specialists for two posts. There was also turnover of Provincial Government staff working on the project, for instance, the Provincial Government Project Manager (PIMPRO) changed three times in East Java. This turnover caused problems with continuity and the maintenance of project implementation capacity. The main contributions to it were the difficulties experienced with the provision of funds to the provinces especially in the early part of the project and the temporary nature of the employment that created a push for key staff to secure permanent employment with a regular income. Project Financing Issues 11. Problems caused by delays in the provision of project funds. Serious operational difficulties were caused especially during the first half of the project by (a) bottlenecks that impeded the flow of Bank funds to the provinces through the SPABP mechanism; and (b) the failure by some provinces to

53 -41 - make counterpart funding available sufficiently early during the fiscal year. These delays meant that there were long time lags of about eight and ten months in the payment of PIU staff salaries during the first two years of the project and various informal measures were taken to alleviate the situation. 12. In West Java, the then PIU Director gave an interest-free loan to the staff to cover part of their salaries and operating expenses, that was repaid when the funds became available. In East Java, all staff worked without being paid until the money arrived. The problem in North Sumatra was handled by taking out a commercial loan to cover salaries of lower level staff and other costs and staff were required later to repay the borrowed money and service the interest costs. 13. Refunds of fees by training providers. In West Java training providers were systematically asked to make a partial refund of fees to the PIUs and CGTUs to cover some operational costs. The justification put forward for this by the PIUs was that they directly provided some services that the training provider was paid for because of the restraints of the contract, for instance, certification, training materials and photographic evidence to show attendance of participants at training. Thus, according to the PIU, the cost of these services had to be recouped from the training providers. In the same way, the CGTUs felt justified in recouping the costs of conducting training needs analysis for their member companies and the associated travel costs from the fee paid to the training providers. The PIU and PIMPRO stated verbally without providing written documentation that the amounts returned were about 3% of the contract amount. However, informal information obtained by the mission without written documentation suggests that the amounts were much greater. The PIU in East Java stated that they did not engage in this practice and that if money was paid by the training providers it was done secretly. 14. In North Sumatra, training providers were partially paid from a commercial loan because of the delay in the flow of funds. This money was repaid with interest when project funds became available. 15. It is the view of the mission that in the cases of the certification and the provision of training materials for participants, the intention was good but the method chosen to deal with the matters was wrong. The operating procedures should have been changed to avoid the unacceptable practice of recouping money in this way and all funds should have been subject to financial control. The recoupment of money by the CGTUs for training needs analysis and travel was clearly wrong as these items were to be covered within their budget. 16. Money returned by the PIU to the Provincial Government. The mission was informed verbally without written documentation that depending on the province, I to 3% of funds given to the project was given back to the Provincial Government in return for services provided by them to the project such as official stamping of documents and photocopying. Also, it was stated verbally without written documentation that the Treasury Office (KPKN) in each province got back about 1% of the loan funds it paid out for services provided. No representative from KPKN was present when this statement was made and the mission did not independently verify that with them. The mission received informal information without documentary evidence, that the 'cost of money' was actually higher in some provinces than the amount advised to the mission. 17. Contributions by the private sector to the cost of training. A key element of the design of the project was the contribution by companies to the cost of training on an upward scale of 20%, 40% and 70% per year of participation in the project. In practice, up to the 40% level was widely achieved but there was resistance to the 70% level that may have been partly caused by the economic downturn. The payments were made in cash or in-kind in the three provinces with the proportion differing between them. The contribution to the SETAS program for small companies was mainly in-kind by the provision of

54 items such as facilities and food. The large companies in the ITS program paid in cash or in-kind through the provision of facilities and instructors. The contribution to the CGTU program by the medium sized companies was mostly in cash. 18. In East Java, the contribution from the private sector was placed in the control of the PIMPRO and was accounted for. In West Java and North Sumatra cash contributions from companies were paid directly to training providers, it did not flow through the PIU though it was administratively recorded by them. 19. It is possible to draw a number of definite conclusions from these findings. The first is that there was a large amount of money circulating in the project that was handled informally (the money involved in the practices in paras. 12, 13, 14, 16 and 18 was not accounted for). The amount of money is unknown but it is safe to conclude that it was substantial. It cannot be estimated as (a) proper financial control was not kept over most of it, and (b) it has not been included in the audits. Another conclusion is that some of the reasons for these informal practices were valid but that many were not. The valid practices should have been formally dealt with by modifying the operating procedures. Conflict of Interest 20. This issue was investigated during the previous review missions (1996, 1997 and 1998) and a result of a letter from the Bank to MOM, different changes were made to project staff in all three project provinces. This mission investigated the causes of the concentration in the award of training contracts in all three provinces and it was found that they there were acceptable and non-acceptable reasons involved. 21. As described by previous missions, there were serious delays in the payment of training providers during the first three years of the project. These were mainly caused by two factors: (a) the flow of funds to the provinces was severely retarded in the first half of the project period as described above, and (b) there were serious bureaucratic problems in the Provincial Government that were only overcome in the third year of the project. There were no serious problems with delays in payment during the last year of the project. Many firms would not tender for training contracts because of this delay in being paid during the first three years of implementation. Also, large firms did not consider the budget to be high enough to make it commercially viable for them to take part in the program. 22. As a result of these constraints, the PLUs felt strongly that they had no choice but to use those training providers that would work under those conditions. Another factor that resulted in concentration was the insistence by large companies in the ITS program to use training providers nominated by them. 23. An unacceptable reason that influenced concentration of training providers was nepotism. This is defined here to cover contracts being given to training providers in which project staff, relatives and friends had an interest. There were serious examples of this in all three provinces and this issue was dealt with by MOM as a result of the letter from the Bank referred to in para. 20. The PIUs claim and the evidence supports this, that the problems that caused the concentration in the award of training contracts were overcome during the last year of the project (the extension) as there was a greater spread of training providers used. Other Staffing Issues

55 There were two instances where relatives of PTFB/PIU staff were employed in the project in North Sumatra. This did not seem to happen in West Java. In East Java there were two cases, one a relative of a PIU staff member and the other a relative of a MOM official. 25. There were a number of cases where PIU and CGTU staff had more than one job even though they were paid to work fully for the project. The problem was most prevalent in East Java and in North Sumatra where five PIU staff had other jobs or businesses. Sustainability of Project Activities Provincial Component. 26. Policy of the Ministry of Manpower. In the three provinces employer organisations and provincial government supported a continuation of the project programs in various modified forms. Each province submitted a proposal for funding the operational phase to the Directorate of Manpower Planning and Development in MOM as this was the directorate responsible for the implementation of the SDP. This directorate prepared a consolidated proposal that it supported for funding in the next financial year beginning April However MOM decided that the proper location for these activities should be the Directorate of Training and Productivity and the proposal was given to them for consideration. The mission was informed that the policy of the Ministry is to support the sustainability of the Skills Development Program in the three project provinces from June of the financial year 1999/2000, and to support replication to other provinces starting from the following financial year. 27. While MOM decided that the program was worthy of support it would not be given funding to cover all costs for example, the salaries of PIU staff. As priority is being given to social safety net programs, the SDP would possibly be facilitated in accessing the Foreign Workers Fund (fees from the licensing of foreign workers to work in Indonesia) from June 1999 through a budget prepared at national level. The Ministry wished to provide funds from this source from April 1999 but the fund was depleted. MOM is also encouraging them to access the social safety net funds through the MOM Provincial Offices and to seek funding from the provincial offices of BAPPENAS. MOM agreed to write to the Skills Development Programs and their provincial offices in the three provinces to help them to access funds. It is understood that support can be given for the overhead cost of running the SDP that would assist the staffing and operational expenses of CGTUs and the operational expenses of offices for the PIUs. 28. Three main issues arise from this decision. The first is that it will not ensure that one of the key activities of the program will be achieved i.e. the encouragement of industry to participate in training and to establish a training system. PIUs that are solely involved in surviving as training providers will not be able to afford to encourage companies to train their staff to meet changing market needs. A promotional activity backed strongly by the employer organisations could be beneficial to companies to prepare for and participate in the forthcoming economic upturn if it encourages them to train to meet market demand. Public encouragement and support for companies to train their workers is a key policy of many OECD countries. 29. The second is one of timing as the PIUs and CGTUs are not likely to be viable in the present economic conditions and therefore, the SDP could slowly die due to lack of funding. In the present economic downturn, only exporting companies could pay the full cost of training and even if they cooperated the objective of the programme would not be achieved as it aims to assist the range of companies from small to large to train their staff in line with market requirements.

56 The third key issue is that there will not be a subsidy to pay the salaries of the PIU staff under this decision. The PIU is the cornerstone for the continued operation of the programn and it is unlikely the program will continue unless the PIU has sufficient qualified staff. For this operational phase, the PIUs have decided to reduce their staff numbers to about four to six which is less than half of the number they had during the project. The three PIUs are trying to find a solution to this key issue at this time. 31. Following the decision of MOM not to provide comprehensive funding to the three project provinces for the operational phase, the project institutions in each province have prepared a fall-back strategy for survival based mainly on charging for their services. The proposals for sustainability in the three provinces show variations from each other and these are described below. 32. West Java. A not for profit organisation 'Training Foundation for Industrial Society' (TRAFFIS) has been established by the PIU. It is providing (a) demand-led training for businesses and (b) counselling and consultancy services for businesses and job seekers that would also include the assistance for business incubators. It is generating commercial income and trying to obtain funding that is consistent with its objectives from other donors, and government ministries including the Ministries of Manpower, Small Businesses and Cooperatives, and Industry. They have the full support of KADIN while the Provincial Government is endorsing the program but is not in a position to give financial support. The PIU currently has four professional staff and it has an office with good equipment and communication facilities including The timing is bad as the economic crisis is making it more difficult for them to earn commercial income. Exporting firms can afford to pay for services but ones engaged in the domestic market are less willing to pay for training at present. Nevertheless, the PIU expects that these firms would be willing to pay 40 to 50% of the total cost of training. There are ten CGTUs operating on a part-time basis while awaiting access to funds from MOM and other government programs. 34. East Java. The position is generally similar to West Java with the important distinction that the surviving project institutions in this province are determined to be viable from earned income. Any support they might get from government programmes would be welcome but they plan to survive without it if necessary. They have been active in generating income since the end of the project. Their services consist of training, consultancy and seminars. The PIU has formed a foundation and they have been provided with a good office by KADIN. There are three professional and three support staff in the former PIU which is less than half of the number under the project. There are eleven active CGTUs and three active small enterprise group training units (SEGTUs) that are trying to be self sufficient. These units have about half of the staff they had under the project. 35. North Sumatra. The establishment of a 'Provincial Body for the Development of Work Skills (BPKK) has been proposed and this is under review for approval and support by the Provincial Government. A decision is expected in April This proposal emerged from a seminar held in the province to explore the sustainability of the project and it has the support of KADIN and individual companies. It is hoped that a combination of provincial government funding, training program funding from MOM and private sector payments for services would ensure sustainability. In the meantime, the PIU and thirteen CGTUs (who are also considering becoming training providers) are operating on a parttime basis. Central Component.

57 Staff Training and Development. During implementation, it was decided that this activity was valuable to the Ministry and therefore, the number of staff studying for masters degrees in Indonesia was increased using government funds. Sufficient funds have been included in the budget for 1999/2000 to complete existing programs and to start 20 more people. The selection of the candidates is based on administrative criteria (administered by MOM) and academic criteria (specified by public universities). 37. Local Labor Market Information System. The full range of activities that were successfully completed during implementation are being sustained in the operational phase using the development budget. These include the: consolidation of data bases in MOM, computer network, intranet facility, development of the MOM home-page on the internet, and a range of improved labor market publications. 38. Young Professional Entrepreneurship Development Program. The scope of the two main activities in this subcomponent were changed substantially by the Government during project implementation. The medium-term training model was abandoned while the short-term model was taken out of the project and implemented nationally using the government budget. The latter model is being sustained by the Government.

58

59 0 ~~~~~~~~~papua NEW GUINEA 8 N I- - -z a 88 u-i Lii $ ' ''r"< <Z -2 Z - Z(L >~~~~~~( U _ <80 z z 2 El Ca _n~~~~~~~~~~~~8 >< X 82R2 Nj~~~~~~~~z CG) ~ ~ ~ ~ ~ 0 88 Z 0~~~~~~~~ Z t Js'v4t.-o 9 & >. "- 0 <Q0bAS,,1,Y,,~~~~~~~Z O~~~~~~~~rla Hz1z z- a ki z z z tm~~~~~~~~~~~~~~~~5 Jz t X s i~~~~~~~~aa -~~~~~g