IMPLEMENTATION COMPLETION AND RESULTS REPORT

Transcription

1 Public Disclosure Authorized The World Bank Document of The World Bank FOR OFFICIAL USE ONLY Public Disclosure Authorized Report No: ICR IMPLEMENTATION COMPLETION AND RESULTS REPORT Loan No ID ON A LOAN Public Disclosure Authorized IN THE AMOUNT OF US$65 MILLION TO THE REPUBLIC OF INDONESIA FOR THE STATISTICAL CAPACITY BUILDING-CHANGE AND REFORM FOR THE DEVELOPMENT OF STATISTICS (STATCAP-CERDAS ( P Public Disclosure Authorized Governance Global Practice East Asia And Pacific Region November 29, 2017 This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. Page 1 of 72

2 CURRENCY EQUIVALENTS (Exchange Rate Effective Nov 1, 2017 Currency Unit = IDR 13, = Indonesian Rupiah (IDR US$1 US$0.74 = IDR 10,000 FISCAL YEAR July 1 - June 30 Regional Vice President: Victoria Kwakwa Country Director: Rodrigo A. Chaves Senior Global Practice Director: Deborah L. Wetzel Practice Manager: Robert R. Taliercio Task Team Leader(s: Noriko Toyoda, Erwin Ariadharma ICR Main Contributor: Lina Lo Page 2 of 72

3 ABBREVIATIONS AND ACRONYMS ABS AIPEG Bappenas BPK BPKP BPS BR CAPI CERDAS CSI EA GDP GDRP GFMRAP GSBPM HR HRMIS IAS ICR ICT IEG ISR IT KemenPAN-RB KPI LBU LKPP M&E MOF MOU NSDS NSO PDO PHRD PINTAR PMO PMU PPI REOI RFP SBFA SOP STATCAP SUSENAS TA Australian Bureau of Statistics Australia Indonesia Partnership for Economic Governance Ministry of National Development Planning (Badan Perencanaan Pembangunan Nasional Supreme Audit Institution (Badan Pemeriksa Keuangan Finance and Development Supervisory Agency (Badan Pengawasan Keuangan dan Pembangunan National Statistics Agency or Statistics Indonesia (Badan Pusat Statistik Business Register Computer-Assisted Personal Interview Change and Reform for the Development of Statistics Corporate Statistical Infrastructure Enterprise Architecture Gross Domestic Product Gross Domestic Regional Product Government Financial Management and Revenue Administration Project Generic Statistical Business Process Model Human Resources Human Resources Management Information System Internal Audit System Implementation Completion and Results Report Information and Communication Technology Independent Evaluation Group Implementation Status and Results Report Information Technology Ministry for Administrative and Bureaucracy Reform (Kementerian Pendayagunaan Aparatur Negara dan Reformasi Birokrasi Key Performance Indicator Large Business Unit National Public Procurement Agency (Lembaga Kebijakan Pengadaan Barang Jasa Pemerintah Monitoring and Evaluation Ministry of Finance Memorandum of Understanding National Strategy for Development of Statistics National Statistics Office Project Development Objective Policy and Human Resources Development Project for Indonesian Tax Administration Reform Project Management Office Project Management Unit Producer Price Index Request for Expression of Interest Request for Proposal Statistical Business Framework and Architecture Standard Operating Procedure Statistical Capacity Building National Social Economic Survey (Survei Sosial Ekonomi Nasional Technical Assistance Page 3 of 72

4 TOR TTL UNECE USDA-NASS Terms of Reference Task Team Leader United Nations Economic Commission for Europe United States Department of Agriculture-National Agricultural Statistics Service Page 4 of 72

5 TABLE OF CONTENTS DATA SHEET... 6 ICR EXECUTIVE SUMMARY I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE II. OUTCOME A. RELEVANCE OF PDOs B. ACHIEVEMENT OF PDOs (EFFICACY C. EFFICIENCY D. JUSTIFICATION OF OVERALL OUTCOME RATING E. OTHER OUTCOMES AND IMPACTS (IF ANY III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION B. KEY FACTORS DURING IMPLEMENTATION C. MAIN EVENTS/FACTORS LEADING TO CANCELLATION IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME.. 30 A. QUALITY OF MONITORING AND EVALUATION (M&E B. ENVIRONMENTAL SAFEGUARD, SOCIAL SAFEGUARD, AND FIDUCIARY COMPLIANCE C. BANK PERFORMANCE D. RISK TO DEVELOPMENT OUTCOME V. LESSONS AND RECOMMENDATIONS ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ANNEX 3. PROJECT COST BY COMPONENT ANNEX 4. EFFICIENCY ANALYSIS ANNEX 5. BORROWER S COMPLETION REPORT AND/OR COMMENTS ANNEX 6. SUPPORTING DOCUMENTS ANNEX 7. RATINGS OF INDICATORS BY SUB-OBJECTIVES Page 5 of 72

6 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P DATA SHEET BASIC INFORMATION Product Information Project ID P Country Indonesia Original EA Category Project Name STATISTICAL CAPACITY BUILDING-CHANGE AND REFORM FOR THE DEVELOPMENT OF STATISTICS (STATCAP- CERDAS ( P Financing Instrument Specific Investment Loan Revised EA Category Not Required (C Organizations Borrower Ministry of Finance Implementing Agency BPS-Statistics indonesia Project Development Objective (PDO Original PDO To increase the effectiveness and efficiency of the Indonesian National Statistics Office (BPS-Statistics Indonesia to produce and disseminate reliable and timely statistics in accordance with international standards and best practices, and to be responsive to user needs Page 6 of 72

7 FINANCING Original Amount (US$ Revised Amount (US$ Actual Disbursed (US$ World Bank Financing TF , , ,438 IBRD ,000,000 8,455,448 8,455,448 Total 65,350,000 8,671,886 8,671,886 Non-World Bank Financing Borrower 14,000,000 8,370,000 1,620,000 Total 14,000,000 8,370,000 1,620,000 Total Project Cost 79,350,000 17,041,886 10,291,886 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 07-Apr Jul Dec May-2017 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s 15-Apr-2014 Amount Disbursed (US$M Key Revisions 0 Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s Cancellation of Financing KEY RATINGS Outcome Bank Performance M&E Quality Unsatisfactory Moderately Satisfactory Modest RATINGS OF PROJECT PERFORMANCE IN ISRs No. Date ISR Archived DO Rating IP Rating Actual Disbursements (US$M Apr-2012 Moderately Satisfactory Moderately Satisfactory 0 Page 7 of 72

8 02 25-Dec-2012 Moderately Unsatisfactory Moderately Unsatisfactory Nov-2013 Unsatisfactory Unsatisfactory Jun Dec Jun Jan Dec-2016 Moderately Unsatisfactory Moderately Unsatisfactory Moderately Unsatisfactory Moderately Unsatisfactory Moderately Unsatisfactory Moderately Unsatisfactory 0 Moderately Unsatisfactory.30 Moderately Unsatisfactory 1.13 Moderately Unsatisfactory 2.32 Moderately Unsatisfactory May-2017 Unsatisfactory Unsatisfactory 6.62 SECTORS AND THEMES Sectors Major Sector/Sector (% Public Administration 100 Central Government (Central Agencies 100 Themes Major Theme/ Theme (Level 2/ Theme (Level 3 (% Page 8 of 72

9 Economic Policy 38 Fiscal Policy 19 Fiscal sustainability 5 Public Expenditure Policy 5 Tax policy 5 Subnational Fiscal Policies 4 Macro-financial policies 15 External Finance 5 Monetary and Credit Policies 5 Macroeconomic Resilience 5 Economic Growth and Planning 4 Migration, Remittances and Diaspora Engagement Private Sector Development 18 Jobs 4 Youth Employment 4 Enterprise Development 14 MSME Development 14 Public Sector Management 43 Public Administration 14 Municipal Institution Building 14 Data Development and Capacity Building 29 Data production, accessibility and use 29 Human Development and Gender 4 Labor Market Policy and Programs 4 Labor Market Institutions 4 4 ADM STAFF Role At Approval At ICR Regional Vice President: James W. Adams Victoria Kwakwa Country Director: Stefan G. Koeberle Rodrigo A. Chaves Page 9 of 72

10 Senior Global Practice Director: Vikram Nehru Deborah L. Wetzel Practice Manager: Shubham Chaudhuri Robert R. Taliercio Task Team Leader(s: ICR Contributing Author: Vivi Alatas Noriko Toyoda, Erwin Ariadharma Lina Lo Page 10 of 72

11 ICR EXECUTIVE SUMMARY The Indonesia Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS Project was approved in April 2011 with the objective of increasing the effectiveness and efficiency of the Indonesian National Statistics Agency or Statistics Indonesia (Badan Pusat Statistik, BPS to produce and disseminate quality statistics. The Government requested a STATCAP loan to support the BPS reform agenda, to enable the organization to better respond to challenges arising from the changes in its external environment. The rapid decentralization process had created a great demand for disaggregated statistics while at the same time reducing the availability of subnational data through line ministries. This had a major impact on BPS. Other challenges for BPS included maintaining capacity to capture new forms of economic activity and continuing to improve cooperation with data producers, users, and respondents. Furthermore, the President had put Bureaucracy Reform as one of the highest priorities of his administration, calling for government agencies to provide services that were better, faster, and cheaper. The theory of change underlying the STATCAP-CERDAS project was that improved people, process, and technology aspects within BPS would significantly increase the quality of the statistical value chain from identifying user needs to producing and disseminating statistics which in turn would contribute to strengthening evidence-based policy making and better governance. The original project cost was US$83 million, of which US$65 million was an IBRD loan and US$18 million was the Government s counterpart budget. The project was to be completed within five years by the end of The project became effective in July 2011, but the Chief Statistician who had initiated and championed the STATCAP-CERDAS preparation was promoted to another ministry within three months, and a new Chief was not appointed until February During this time, BPS started the procurement of the project management (Owner s Agent and statistical consultancy packages, but progress stalled in the transition. Implementation was additionally hampered by the loss of the Project Management Unit (PMU structure due to regulatory changes in December Implementation continued to be delayed through 2012, and the project restructuring dialogue was formally initiated in January STATCAP-CERDAS was restructured in April 2014 to align the scope with the new leadership s direction, focusing on the improvements to the core statistical business processes and systems of BPS and reducing scope where progress had been made by BPS under its own financing or with other technical assistance (TA. The implementation of the back-office information technology (IT systems (Human Resources Management Information System [HRMIS] and Internal Audit System [IAS] was dropped, which contributed to the bulk of the partial cancellation amount. The financing amount was reduced accordingly to US$47 million of IBRD loan and US$14 million of counterpart budget, and the closing date was extended to November 30, STATCAP-CERDAS gained some traction after restructuring, as key consultancies were contracted and all project components moved into implementation stage. The Statistical Transformation and Quality Improvement consultant and the IT Strategy and Design consultant developed inputs to the technical specifications for the design, development, and implementation of the Integrated Statistical Applications Systems package. The Human Resources (HR Management and Organization Design consultant also produced people and organizational alignment requirements based on the new business processes and functions. Implementation of the integrated statistical information systems was critical to providing Page 11 of 72

12 common IT tools and applications to support statistical production and dissemination processes across BPS. The Request for Expression of Interest (REOI for the statistical IT systems package was launched in February 2016, and the technical evaluation was completed by the end of October However, due to the complexity and size of the package, BPS requested the Government s internal audit agency (Finance and Development Supervisory Agency [Badan Pengawasan keuangan dan Pembangunan, BPKP] to review the procurement process before submitting the Technical Evaluation Report to the World Bank. BPKP raised a number of findings, notably that the procurement committee had not obtained the prior approval of the cost estimates from the competent authority in the Government before initiating the selection process and that the short-listing process was not in compliance with the World Bank s Consultant Guidelines, even though the World Bank had issued no objections after having prior reviewed the process, and in spite of constant efforts from the procurement team to address the concerns of the client while reassuring that process was in compliance with the Consultant Guidelines. Due to further delays, the project was suspended in December 2016 with remedial actions to (a award the contract for the Design and Implementation of Integrated Statistical Information Systems consultancy and (b establish an appropriate counterpart project team for its implementation. BPS failed to complete the procurement, and the project was eventually cancelled in May 2017 with US$8.46 million, or 13 percent of the original loan amount, disbursed. Throughout its lifetime, project implementation was affected by turnover in key counterparts at the leadership as well as operational levels and suffered from challenges in isolating staff from the businessas-usual to work on the reform project. STATCAP-CERDAS was led by three Chief Statisticians throughout its life cycle. All three chiefs embraced the same to be vision for BPS modernization but differed in views on the path toward the to be state. The original scope of the STATCAP-CERDAS presented a comprehensive institutional transformation program, which touched all parts of the BPS organization and required a leader to drive this multipronged transformation across the rank and file of his/her organization. The PMU was also reconstituted four times, partly due to rigid constraints of the civil service regulation to establish a cohesive structural unit to manage the reform on a full-time basis. Although the project was not successful in deploying the information system to produce and disseminate statistics, STATCAP-CERDAS provided BPS with the requisite tools to continue its modernization. The TA provided by the project supported the design of the BPS Statistical Business Framework and Architecture (SBFA with standardized business processes and operating model, the detailed designs of various Corporate Statistical Infrastructure (CSIs, IT Strategy and Enterprise Architecture (EA, and macroorganization redesign. The TA provided by the World Bank through trust fund support as well as by development partners such as the Australian Bureau of Statistics (ABS was aligned with the STATCAP- CERDAS framework, which contributed to improved results. At the time of the Implementation Completion and Results Report (ICR, BPS had secured state funding from the Ministry of Finance (MOF over the next four years to continue its transformation program and implement the statistics information systems development. The overall outcome rating for the project is Unsatisfactory, largely due to the project s inability to complete implementation. The rating for relevance is High, with the ongoing need for the project remaining strong. The rating for efficacy is Modest, with some of the Project Development Objective (PDO indicators improved, though most of the intermediate results indicators underperformed. The Page 12 of 72

13 rating for efficiency is Negligible, due to considerable implementation delays and early cancellation. The World Bank performance overall is Moderately Satisfactory. In this ICR, the team has presented lessons and recommendations related to planning for sustainability, championing reform, organizing for reform, planning for complex information and communication technology (ICT procurement, reorienting auditors focus, and targeting niche markets, which may be applicable to other institutional reform initiatives. Page 13 of 72

14 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. At the time of project preparation in 2011, Indonesia had emerged as a middle-income country with gross domestic product (GDP per capita at US$2,230 and a resilient economy that weathered the 2008/09 global financial crisis better than its neighbors. Although the poverty rate had been decreasing steadily, the rate of poverty reduction had not kept pace with economic growth. The 2001 big-bang decentralization, with most of the public service delivery responsibilities and corresponding planning and budgeting authorities devolved to the subnational levels of governments, increased the demand for relevant, accurate, timely, and disaggregated data. 2. Indonesia s National Statistics Agency or Statistics Indonesia (Badan Pusat Statistik, BPS is a central government agency, organized vertically with regional offices located in all provinces and districts across the archipelago. BPS is the coordinator of Indonesia s National Statistics System and is responsible for producing a wide range of key social and economic statistics, including poverty, labor, price, trade, production, and national accounts data. Official statistics are a crucial element of informed, evidence-based decision making and monitoring of national development strategies. Government agencies from the Ministry of National Development Planning (Badan Perencanaan Pembangunan Nasional, Bappenas to the Ministry of Finance (MOF to line ministries depend on BPS statistics for planning and budgeting, targeting of policies and programs, and outcome monitoring and evaluation (M&E. 3. At appraisal, the leadership of BPS was keenly aware of the changes in its external environment and was committed to a comprehensive reform of BPS to deliver better results for the public. The rapid decentralization process in Indonesia that started in 2001 had created a great demand for subnational statistics while at the same time reducing the accessibility of subnational data through the line ministries. This had a major impact on BPS. Other challenges for BPS included maintaining capacity to capture new forms of economic activity, and continuing to improve cooperation with data producers, users, and respondents. Much of the BPS survey operations still relied on point-to-point, paper-based processes, and its technology infrastructure was outdated; so, there was significant room for efficiency gains through investments in process improvements and technology. There were also opportunities for greater integration of surveys and greater use of common tools and shared systems across the organization to improve the timeliness and efficiency of data collection and processing. Furthermore, the President had put Bureaucracy Reform as one of the highest priorities of his administration and called for government agencies to provide services that were better, faster, and cheaper. 4. The World Bank had provided various technical assistance (TA to BPS, including support to develop its National Strategy for Development of Statistics (NSDS, which identified the need for BPS to respond to challenges of increasing demand for more data arising from rapid decentralization and the country s changing socioeconomic environment. In light of escalating demands and limited resources, BPS requested the World Bank for Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS financing to assist its efforts in improving the quality of major statistical collections and investing in core statistical infrastructure to shift the organization toward more corporately integrated and coherent operations. The Chief Statistician also used STATCAP-CERDAS as the framework for harmonizing development partner assistance, including its cooperation program with the Australian Bureau of Statistics (ABS, as outlined in the Memorandum of Understanding (MOU between the two National Statistics Offices (NSOs. Page 14 of 72

15 5. STATCAP-CERDAS was aligned with the priorities of the World Bank s Country Partnership Strategy (Report No IND under the cross-cutting engagement to support central government institutions and systems. The project was to strengthen BPS s institutional capacity do deliver better information and services to the Government and the public. The project also aimed to support the national development priorities of bureaucracy and governance reform through (a improved statistics from BPS to support informed and evidence-based decision making across the Government and (b improving the institutional performance and efficiency of BPS. Theory of Change (Results Chain 6. The theory of change underlying the STATCAP-CERDAS project was that improving the people, process, and technology aspects within BPS would significantly increase the quality of the statistical value chain from identifying user needs to producing and disseminating statistics which in turn would contribute to strengthening evidencebased policy making and better governance. Figure 1. Theory of Change for STATCAP-CERDAS Project Project Development Objectives (PDOs 7. The STATCAP-CERDAS project was to increase the effectiveness and efficiency of the Indonesian National Statistics Office (BPS-Statistics Indonesia to produce and disseminate reliable and timely statistics in accordance with international standards and best practices, and to be responsive to user needs. Key Expected Outcomes and Outcome Indicators 8. The PDO contained several compound objectives and was unpacked into two sub-objectives for this Implementation Completion and Results Report (ICR (see Figure 1: (a To increase the effectiveness of BPS in producing and disseminating quality statistics (b To increase the efficiency of BPS in producing and disseminating quality statistics Page 15 of 72

16 9. Because STATCAP-CERDAS was to support fundamental reforms and change efforts to improve the quality of statistics, key PDO-level indicators were arranged along the quality dimensions of official statistics, as commonly defined by international organizations and NSOs. The indicators measured improvements in accuracy, relevance, timeliness, accessibility, coherence, and interpretability of key BPS statistics. Components 10. STATCAP-CERDAS comprised five components as follows (see also Figure 1: A. Improved Statistical Quality (US$32.8 million to streamline and integrate its statistical business processes B. Improved ICT Platform and Statistical Information Management Systems (US$30 million to modernize its IT systems and infrastructure C. Improved Human Resources Management and Development (US$9.3 million to strengthen its human resources (HR policies and practices from personnel administration toward people management D. Improved Organizational Alignment (US$4.8 million to restructure its organization in line with new business processes and functions E. Project Management (U$6.1 million to ensure integration across components and manage change across the organization 11. The original project cost was US$83 million, of which US$65 million was IBRD loan and US$18 million was the Government s counterpart budget. The project was to be completed in five years by the end of B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE Revised PDOs and Outcome Targets 12. After two years of zero disbursement, a Level 2 restructuring of the project was completed in April 2014 to focus the project scope on improving BPS s core statistical business and to account for improvements being implemented by BPS and through other TA. The loan amount was reduced from US$65 million to US$47 million, and the counterpart budget from US$18 million to US$14 million. The project closing date was extended to November 30, The PDO remained the same after restructuring, while the project was streamlined to focus on improvements to BPS s core business processes and systems for statistical production and dissemination. A large part of the reduction in the financing amount stemmed from the cancellation of design and implementation of two information systems Human Resources Management Information System (HRMIS and Internal Audit System (IAS which could be the next generation of reform initiatives for BPS. Page 16 of 72

17 Revised PDO Indicators 14. The desired outcome remained as improving statistical quality. The PDO indicators were streamlined but continued to reflect the statistical quality dimensions accuracy, relevance, timeliness, accessibility, coherence, and interpretability. The intermediate results indicators were also revised and streamlined to align with the scope adjustments and reductions under each component. Table 1 summarizes the original and revised indicators. Table 1: Original and Revised Indicators Indicators Action per Restructuring PDO INDICATORS PDO 1: Accuracy: Demonstrable improvement in the accuracy of key statistics as evidenced by (a A reduction in the statistical discrepancy between the quarterly and Dropped annual production and expenditure estimates of growth in the national (TA on National Accounts provided by ABS accounts (b An increase in the response rate of the Annual Survey of large and Unchanged medium manufacturing businesses (c An increase in the user satisfaction with the accuracy of BPS statistics, including the extent of revisions Revised An increase in the user satisfaction with the accuracy of BPS statistics PDO 2: Relevance: Availability of important new statistics as evidenced by the establishment of (a The introduction of the 2008 version of the System of National Dropped Accounts (TA on National Accounts provided by ABS (b A national producer price index Unchanged (c Quarterly labor force statistics Dropped (BPS changed its internal target. PDO 3: Timeliness: Improvements in the timeliness of key statistics as evidenced by (a Reducing the time lag in the production of preliminary and final Dropped imports and exports data (BPS already achieved timeliness improvement. (b Reducing the time lag in the publication of final figures from the Dropped National Social Economic Survey (SUSENAS (BPS would continue to improve in this area on its own. (c Reducing the time lag in the publication of annual labor force Unchanged estimates PDO 4: Accessibility: An improvement in the accessibility of BPS statistics as evidenced by (a A dynamic up-to-date website Unchanged (b An increase in the user satisfaction with the accessibility of BPS Unchanged statistics PDO 5: Coherence: An improvement in the coherence of BPS statistics As evidenced by a reduction in the discrepancy between the sum of the Unchanged Provincial estimates of GDRP and the GDP for Indonesia PDO 6: Interpretability: An improvement in the interpretability of BPS statistics as evidenced by (a An increase in the statistics produced by BPS, especially widely used statistics (for example, national accounts, prices, labor, and SUSENAS that have supporting meta data that describe the concepts, data sources, and statistical methods that have been used (b The availability of seasonally adjusted monthly and quarterly statistics Unchanged (slightly revised An increase in the statistics produced by BPS that have supporting meta data Dropped (BPS would continue to improve in this area on its own. Page 17 of 72

18 INTERMEDIATE RESULTS INDICATORS Component A: Improved Statistical Quality 1. User groups and technical advisory groups are established Dropped (Dropped due to process-oriented nature of the indicator 2. Successful implementation of a Quality Assurance Framework across Unchanged BPS 3. The establishment of a Common Business Register (BR and a Large Unchanged Business Unit (LBU 4. A significant upgrade in the source data used in the compilation of the Dropped National Accounts (TA on National Accounts dropped from scope 5. Availability of an integrated data and information service system Unchanged supported by the data warehouse Component B: Improved ICT Platform and Statistical Information Management Systems 6. Improvement in common ICT infrastructure and its performance Dropped (Streamlined with other ICT indicators 7. ICT standardization (ICT governance, ICT architecture, ICT Revised procurement, applications development methodology, version Improvement in ICT Governance and Management management of applications and software, systems testing, standard tools, and support 8. Improved availability of suitably qualified ICT experts Dropped (Streamlined with other ICT indicators Added Development of application systems for statistical production and dissemination Component C: Improved Human Resources Management and Development 9. A workforce that is aligned with BPS s future strategic direction Revised A Human Resource that is aligned with BPS s future strategic direction 10. The successful implementation of an integrated HRM System Dropped (Subcomponent dropped from scope 11. An effective performance management system for BPS Revised An effective, integrated performance management system for BPS Component D: Improved Organizational Alignment 12. Improved alignment of the BPS organization and the future work Unchanged environment 13. BPS functions in accordance with agreed corporate values Dropped (Subcomponent dropped from scope 14. Development of a new monitoring and inspection system Dropped (Subcomponent dropped from scope 15. Effective responses to reviews, audits, and coherency studies Dropped (Subcomponent dropped from scope Component E: Project Management 16. Effective project management Unchanged 17. Procurement carried out in timely and transparent manner Dropped (Streamlined with other project management indicators Note: GDRP = Gross domestic regional product; ICT = Information and communication technology. Page 18 of 72

19 Revised Components 15. Restructured project components continued to be aligned with the three pillars of institutional transformation People (Components C and D, Process (Component A, and Technology (Component B, and were unchanged to deliver the desired outcome. However, subcomponents and activities were revised as follows: A. Improved Statistical Quality (US$6.9 million. This component was reduced in scope to focus on TA for specific statistical product lines and on analysis and design of core statistical infrastructure to enable greater statistical integration. Scope revision took into account improvements made by BPS and through other complementary TA, for example, in National Accounts area. B. Improved ICT Platform and Statistical Information Management Systems (US$29.06 million. This component remained largely unchanged to develop BPS s information technology (IT strategy, Enterprise Architecture (EA, information systems governance framework, standardization and centralization of IT operations and resources, and development of integrated statistical production and dissemination systems. Hardware component was decoupled from the statistical IT applications package to be procured through the Government s counterpart budget. C. Improved Human Resources Management and Development (US$2.3 million for Components C and D, plus US$4.37 million for training and capacity building. This component was reduced in scope to focus on improvements in HR management and development policies and practices, design performance management system, and strengthen BPS s HR function to shift from personnel administration to people management. Financing for training and capacity building of BPS staff was increased substantially, while the subcomponent on upgrading the HRMIS was dropped from the project. D. Improved Organizational Alignment (budget combined with Component C above. This component was reduced in scope to focus on the new organization design in alignment with new business processes and functions. The subcomponent on strengthening the internal audit capacity, including implementation of the IAS, was dropped from the project. E. Project Management (US$4.05 million. A new structural unit to manage BPS transformation and serve as the Project Management Unit (PMU for STATCAP-CERDAS was proposed to the Ministry for Administrative and Bureaucracy Reform (Kementerian Pendayagunaan Aparatur Negara dan Reformasi Birokrasi, KemenPAN-RB in November 2013, and the new structure was established in July This component also financed the Owner s Agent consultant to support project management, change management, and quality assurance. Other Changes 16. Despite scope reduction through restructuring, STATCAP-CERDAS failed to sustain steady progress and the project was cancelled in May 2017 with US$8.46 million or 13 percent of the original loan amount disbursed. Rationale for Changes and Their Implication on the Original Theory of Change 17. The restructuring in April 2014 was intended to align the project with the new organizational direction after project implementation stalled in the leadership transition. The first Chief Statistician was promoted to another agency within the first six months after loan signing, and the new statistician was not appointed until February During this time, BPS started the procurement of the project management (Owner s Agent and statistical consultancy packages, but progress stalled in the transition. Implementation was additionally hampered by the loss of the PMU structure due to regulatory changes in December Implementation continued to be delayed through 2012, and Page 19 of 72

20 the project restructuring dialogue was formally initiated in January BPS initially proposed to reallocate the bulk of the resources to the capacity-building component, as well as reduce the loan amount substantially. Eventually, in discussion with Bappenas and the MOF, it was agreed to keep the PDO unchanged to support the BPS institutional reform, but to focus the scope on core statistical improvements and integration. 18. Project restructuring focused the project scope on improvements to the core business of BPS, that is, production and dissemination of statistics. The original scope of STATCAP-CERDAS presented a comprehensive institutional transformation to improve people, process, and technology elements, including the rollout of three information management systems Integrated Statistical Applications Systems, HRMIS, and IAS which would require the leader to drive this multipronged approach across the rank and file of his/her organization. It was ambitious considering the enormous change in skills, processes, and technologies required to properly operationalize these changes and systems across the country within the original project period of five years. Implementation of backoffice information systems (HRMIS and IAS were dropped at restructuring. Consistent with learning from previous World Bank operations, hardware components were also decoupled from the Integrated Statistical Applications Systems package to simplify the procurement. The World Bank financing was to be for the design, development, and implementation of the information system itself, and BPS would procure the requisite hardware through the Government s counterpart budget. 19. Most of the dropped or revised indicators as a result of the restructuring were adapted to reflect progress made by BPS, as well as through complementary TA provided in part by the ABS. The changes, therefore, did not affect the original theory of change. 20. Although STATCAP-CERDAS gained traction after restructuring, the project was cancelled in May 2017, because BPS failed to meet the conditions for lifting the suspension, that is, (a to sign the contract for the integrated statistical information systems package and (b to establish an appropriate counterpart project team for its implementation. At the request of BPS, the Government s internal audit agency (Finance and Development Supervisory Agency [Badan Pengawasan keuangan dan Pembangunan], BPKP reviewed the procurement process and raised a number of findings, notably that BPS s procurement committee had not obtained prior approval of the estimated cost from the competent authority in the Government before initiating the selection process. BPKP also made findings that the short-listing process carried out by BPS s procurement committee was not in compliance with the World Bank s Consultant Guidelines, even though the World Bank had issued no-objections after having prior reviewed and found the BPS procurement committee s recommendations to be consistent with the principles of the World Bank s Consultant Guidelines. Based on BPKP s findings, BPS proposed to cancel the ongoing procurement process and restart a new process, though the World Bank provided further clarifications on the applicability and interpretation of the World Bank s Consultant Guidelines. Given there were only 18 months left until the project closing date of November 2018, the World Bank proceeded to suspend loan disbursement in December 2016 with remedial actions intended to move the implementation forward. STATCAP-CERDAS was officially cancelled in May 2017, because conditions for lifting the suspension were not met. Page 20 of 72

21 II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating Relevance Rating of PDO: High 21. The project aimed to support BPS s Strategic Plan for Change and Reform for the Development Statistics (CERDAS and institutional transformation, and the PDO remains highly relevant. Official statistics are a crucial element for informed, evidence-based decision making and monitoring of national development strategies. Demand for better quality, more timely, and disaggregated statistics from BPS remains high and continues to increase, as other sources of data such as administrative data systems remain underdeveloped. Within BPS, timeliness and quality improvements are still hindered by lack of common IT tools, for example, a common data collection and processing platform could be used by all survey areas and enable electronic data collection from the field, and corporate data warehouses would consolidate data that currently reside in disparate databases to enable data management across the organization. As of the time of this ICR, BPS has secured state budget over the next four years to continue its reform agenda and implement the statistical production and dissemination information systems. The STATCAP- CERDAS is aligned with the current Country Partnership Framework, contributing to the objective of improving social protection programs with higher quality statistics toward targeting the poor and vulnerable, and World Bank operations would have benefited from better BPS data. 22. The original design of the STATCAP-CERDAS was ambitious in rolling out three information systems Statistical Production and Dissemination Applications System, HRMIS, and IAS within the original project period of five years, given the average time to complete a World Bank project to implement enterprise-wide information system is 7.9 years, with the longest taking over 13 years. Through restructuring, the project scope was realistically adjusted to focus on the design and implementation of the Statistical Applications System to enable the efficient production and dissemination of statistics. At restructuring, efforts were made to strengthen implementation arrangements by establishing a structural unit for statistical transformation to also serve as STATCAP-CERDAS PMU and reconstituting the external advisory board. Page 21 of 72

22 B. ACHIEVEMENT OF PDOs (EFFICACY Efficacy of PDO: Modest 23. The STATCAP-CERDAS project was designed to comprehensively transform the way BPS conducted its business by standardizing and integrating its processes and leveraging IT to improve the quality of statistics. By project cancellation, most of the PDO indicators accuracy, relevance, timeliness, accessibility, coherence, and interpretability had improved. STATCAP-CERDAS also provided the framework for complementary TA notably by the ABS, as well as by the World Bank, which contributed to the statistical quality improvements, despite the project implementation delays and cancellation. Some changes in methodologies, such as the user satisfaction surveys, may affect the direct comparability of achieved results with baseline. 24. The project Results Framework with PDO indicators and intermediate outcomes indicators was reconstructed to strengthen the results chain that supports the project s theory of change. Achievements of the project by effectiveness and efficiency are collected in annex 7 and summarized in Table 2. Table 2: Summary of Achievements by Sub-objectives Indicator Rating according to IEG Guidelines High Substantial Modest Negligible 1. Increased effectiveness of BPS in producing and disseminating quality statistics PDO INDICATORS FOR SUB-OBJECTIVE 1 Total for PDO Sub-objective Percentage of PDO Indicators for Sub-objective INTERMEDIATE RESULTS INDICATORS FOR SUB-OBJECTIVE 1 Total for Intermediate Results for Sub-objective Percentage of Intermediate Results Indicators for Subobjective Increased efficiency of BPS in producing and disseminating quality statistics PDO INDICATORS FOR SUB-OBJECTIVE 2 Total for PDO Sub-objective Percentage of PDO Indicators for Sub-objective INTERMEDIATE RESULTS INDICATORS FOR SUB-OBJECTIVE 2 Total for Intermediate Results Indicators for Sub-objective Percentage of Intermediate Results Indicators for Subobjective Note: IEG = Independent Evaluation Group. Assessment of Achievement of Each Objective/Outcome (a Sub-objective 1: Increased effectiveness of BPS in producing and disseminating quality statistics Rating: Modest 25. Most of the design of the building blocks to transform BPS and increase the quality of Indonesia s statistics were achieved as reflected by the status of the intermediate indicators. Designs of the BPS Statistical Business Framework and Architecture (SBFA; Corporate Statistical Infrastructure (CSIs, such as quality governance structure and statistical business register; and BPS IT Strategy, EA, core managerial competencies, and BPS-wide strategy map and associated key performance indicators (KPIs were completed, with some endorsed (SBFA and EA by the BPS Page 22 of 72

23 executives. These design documents are the basis for the next phase of BPS s transformation journey to be implemented. The following are the achievements of the four PDO-level indicators that contribute to increased effectiveness: Accuracy. Statistics accuracy was improved with response rate to large and medium manufacturing businesses increased from 60 percent to percent (target at 74 percent at project cancellation, and user satisfaction increased from 43 percent to percent, exceeding the target of 75 percent. It should be noted, however, that the methodology for User Satisfaction Surveys conducted by BPS changed over the years, which may affect the direct comparability of the achieved level with the baseline. Relevance. New statistics for national Producer Price Index (PPI were published, with additional service sectors (including accommodation, food and beverage, passenger transportation, electricity and gas supply, and water supply, and studies on further sectors (including freight transportation, telecommunications, financial services, education, health, and waste management have been conducted. For provincial PPI, producer survey data collection has been expanded to all provinces in 2016 and has been prepared for three sectors in five provinces. All these together move BPS a step toward improving availability and coverage of national and subnational PPI but short of the original target of covering all provinces and most large districts. Coherence. The discrepancy between the sum of the provincial estimates of GDRP and the GDP for Indonesia has increased from 0.08 percent to 2.03 percent. BPS adopted methodological changes to eliminate the discrepancy (0 percent in 2010, but the value has been increasing slightly since then. Interpretability. At project appraisal, there was no fully documented metadata. At project cancellation, BPS has established an online Microdata Catalog System, containing metadata information for 310 data sets. Increased availability and accessibility of metadata have contributed to improved interpretability of BPS statistics. (b Sub-objective 2: Increased efficiency of BPS in producing and disseminating quality statistics Rating: Modest 26. With the procurement of the statistical IT systems stopped (equal to more than 50 percent of the restructured loan amount, and none of the other contributing factors advanced to planned targets, achievement of this sub-objective is Modest. Timeliness. The time lag in the publication of annual labor force estimates improved from 150 days to 90 days, short of the target of 60 days. Further reduction in time lag will likely be difficult to achieve without the implementation of Computer-Assisted Personal Interview (CAPI or electronic data collection. Accessibility. A dynamic up-to-date website with BPS statistics was improved with access increased from 51,000 to 127,147 unique visitors per month (monthly average over the year. There was also an increase from 61 percent to percent (target was 80 percent in user satisfaction on the accessibility of BPS statistics. Page 23 of 72

24 Justification of Overall Efficacy Rating 27. On balance, the overall achievement of objectives given above is assessed as Modest. 28. By project cancellation, BPS had the requisite tools to continue its modernization agenda. Despite the failed implementation of the integrated statistical information systems, which could have moved BPS a major step forward toward improving its operational efficiency and statistical quality, the STATCAP consultancies supported to produce key deliverables aimed to lay the foundation, including the standardized business processes based on the Generic Statistical Business Process Model (GSBPM of the United Nations Economic Commission for Europe (UNECE, EA, and detailed design documents for CSIs, to implement the IT systems for the statistical production and dissemination and rollout the reform. C. EFFICIENCY Assessment of Efficiency and Rating Efficiency Rating: Negligible 29. With US$8.46 million, or 13 percent, of the original loan amount disbursed, with modest outcomes overall, the efficiency rating is assessed as Negligible. This also takes into consideration modest to substantial advances in some of the PDO-level indicators and zero disbursement over the initial three-year period. D. JUSTIFICATION OF OVERALL OUTCOME RATING Overall Outcome Rating: Unsatisfactory 30. With zero disbursement pre-restructuring, this ICR does not adopt the split rating approach. 31. The justifications for ratings of relevance, efficacy, and efficiency are elaborated above. Rating for relevance is High, efficacy is Modest, and efficiency is Negligible. Taken together, the overall outcome rating is Unsatisfactory based on IEG guidelines. Table 3. Overall Outcome Rating Relevance High Efficacy Effectiveness Efficiency Modest Modest Modest Efficiency Negligible Overall Outcome Unsatisfactory E. OTHER OUTCOMES AND IMPACTS (IF ANY Institutional Strengthening 32. As stated earlier, the project supported the design of the building blocks to transform BPS and to increase the quality of Indonesia s statistics, which has contributed to strengthening BPS s institutional capacity to continue its transformation. Page 24 of 72

25 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 33. STATCAP-CERDAS was designed to support BPS to implement its Strategic Plan for Change and Reform for the Development of Statistics (CERDAS, which also supported the implementation of the BPS Bureaucracy Reform initiative. The CERDAS plan was developed by BPS through extensive internal and external stakeholder consultations and was part of its medium-term strategy (Rencana Strategis/Renstra that set out BPS s improvement priorities to be implemented over a five-year period. Based on the CERDAS plan, BPS completed a BPS Modernization Proposal in 2010 with detailed reviews of the existing states of statistical methodologies, ICT, and HR. Because the ambitious reform would touch all parts of the BPS organization, the broad-based consultative approach was intended to foster shared vision and ownership within BPS among the leadership and the rank and file of BPS, both in the headquarters and regional offices. 34. The technical design of STATCAP-CERDAS also involved BPS s development partners, such as the ABS, Australia Indonesia Partnership for Economic Governance (AIPEG, Statistics Netherlands, United States Department of Agriculture-National Agricultural Statistics Services (USDA-NASS, and Statistics Korea. In particular, the ABS and BPS have maintained a long-term partnership for the last 20 years that includes peer-to-peer learning and technical advisory support, and its MOU for cooperation was closely aligned to STATCAP-CERDAS to provide complementary TA during project implementation. 35. Policy and Human Resources Development (PHRD project preparation grant was mobilized to support the detailed analysis and development of implementation requirements, but this also resulted in lengthy project preparation period. In addition, it was expected that the procurement documents for the initial year would be finalized with the support of the preparation consultant before effectiveness, but the products delivered by the consultant were deemed not fit for purpose and updated by BPS. By project effectiveness, it had been almost three years since the conception of the project at BPS, and the project champion and initiator, the first Chief Statistician, was promoted to a new post in another ministry. Consequently, this resulted in the loss of reform momentum, because the leadership change occurred shortly after loan effectiveness and time elapsed in the transition. 36. Overall risks at preparation and during implementation were both rated as Medium Driven by Impact. The main risks identified were related to the implementing agency s capacity and governance, including low capacity to manage complex ICT procurement and contracts and low capacity to manage culture change from silo to corporate culture. While the appropriate risks were identified, implementation agency capacity risks were underestimated and were indeed rated High during implementation. Turnover at the top significantly affected the implementation progress, and the change management interventions, as designed, were not able to ensure the stability of the reform momentum through these transitions. 37. At appraisal, BPS demonstrated a readiness to implement its reform agenda and modernization program. Notably, during project preparation, the CERDAS plan had been developed through extensive consultation process involving BPS leaders and the rank-and-file staff in its headquarters and regional offices, as well as external data users; STATCAP-CERDAS was aligned with the broader Bureaucratic Reform design; a detailed requirements analysis was conducted using the project preparation grant, as well as World Bank and donor TA support; and a PMU had been established with full-time staff. However, some weaknesses remained, including the temporary structure of the PMU and lackluster performance of the preparation consultant resulting in incomplete bidding documents by project effectiveness. Page 25 of 72

26 B. KEY FACTORS DURING IMPLEMENTATION Factors Subject to Government Control 38. Turnover in key counterparts affected project implementation progress despite the extensive consultation process during project preparation. The first Chief Statistician led the design and signing of the project, steered the project to effectiveness less than three months after loan signing, and was then promoted to become the Vice Minister for Agriculture. Subsequent leadership changes and transitions across the board at Echelon 1 and 2 levels resulted in implementation delays. Even though officials promoted up the rank had been involved in the preparation of the STATCAP-CERDAS, project implementation progress was sluggish. Shortly after effectiveness, BPS launched the procurement of the Owner s Agent and statistical consultants, but both processes stalled after the short-listing stage. By the time of restructuring, the overall performance rating was Unsatisfactory with all project implementation activities, including procurement processes, put on hold, while project restructuring discussions took place between the World Bank and BPS. 39. Major implementation delays followed each change at the top, despite conscious engagement to involve other Echelon 1 officials and middle management in the reform dialogue and decision-making process. The project governance and management framework did not work optimally during STATCAP-CERDAS implementation. While the internal modernization plan was signed off by unit leads, readiness for reform was mostly lacking in the top management in the headquarters. Two key structures were designed at the pinnacle of the project governance framework, namely, the Steering Committee and the External Advisory Board. The Steering Committee was chaired by the Chief Statistician and comprised all eight Echelon 1 officials of BPS, to provide strategic guidance and decisions. The External Advisory Board was designed to provide a forum for key stakeholders of BPS to become involved in the reform process and guide the progress of STATCAP-CERDAS and was chaired by the Bappenas Deputy Minister, with members from ministries, the central bank, academia, and industry (Indonesian Chamber of Commerce [Kamar Dagang dan Industri Indonesia]. These bodies did not function optimally and often could not meet regularly as planned, due to busy schedules of their members. 40. A lack of a stable PMU and dedicated reform teams for implementing STATCAP-CERDAS also contributed to the slow progress. During the project lifetime, the PMU was reconstituted four times. Given its criticality, submission of the proposal to establish the PMU as a permanent unit Directorate of Statistical Integration and Transformation was included as a covenant in the loan agreement. BPS submitted the proposal to KemenPAN-RB but was not successful in negotiating a new structural unit until July Even then, a new Echelon 2 unit could not be added due to restrictions under the prevailing civil service regulations, so three existing Echelon 3 units were vacated to manage the three transformation areas: statistics, IT, and HR/organization. Only the statistical transformation unit staff reported directly to the Project Management Office (PMO head. In addition, other technical staff were appointed to serve on project working groups and task forces with specific responsibilities, for example, finalizing the SBFA and working with consultant experts to develop the detailed designs for CSIs. However, these arrangements were not effective as staff also had their business-as-usual tasks in practice, and working relationships between BPS staff and consultants were not optimized. 41. STATCAP-CERDAS did not provide sufficiently visible changes and tangible results for the rest of the organization early on, because most of the major process and organizational shifts could not be made until the IT tools and systems were in place first. The World Bank s complementary TA activities also endeavored to address this challenge through providing some concrete demonstration projects (for example, metadata standardization and CAPI pilot, but they were not sufficient. BPS also did not utilize effectively the US$14 million counterpart fund to support activities to increase implementation capacity or to achieve quick wins. This lack of quick wins contributed to waning Page 26 of 72

27 ownership of the reform, despite the change management efforts through the network of change agents throughout the organization. 42. The project gained some traction after restructuring. Four key consultancy contracts were awarded after restructuring, and all project components moved into the implementation stage. With the Owner s Agent on board in December 2014, the Chief Statistician led the relaunching of STATCAP-CERDAS in January 2015 to demonstrate the leadership commitment to BPS staff and external stakeholders. The Bappenas Minister expressed high-level support. Project activities started with the support of TA consultants, including change management and communications activities in headquarters and regional offices; the development of the SBFA with standardized business processes and specification of common statistical infrastructure; and the development of IT strategy, EA, and standard operating procedures (SOPs for IT services. 43. It took one and a half years after project relaunching for BPS to finalize the technical requirements for the integrated statistical information systems development package, the core of the project to deliver the tools and systems required to support the business process and organizational changes. Delays in the selection of the Statistical consultant and its subsequent slow work progress to prepare technical requirements for the information systems package affected the timeline of the procurement. The preparation of the Terms of Reference (TOR for the systems package required inputs from both the Statistical and IT Design consultants and was not finalized until July While the TOR was being finalized, BPS initiated the selection process by issuing the Request for Expressions of Interest (REOI in February 2016 and managed to proceed with the procurement relatively quickly to completing the technical evaluation in less than four weeks (which is a remarkable performance for such a large-value, complex IT contract but failed to submit the technical evaluation report by the end of October 2016 for the World Bank s prior review. This made it impossible to maintain the contract implementation start date in January 2017 and project completion by November 2018 unattainable. Factors Subject to World Bank Control 44. There were five task team leaders (TTLs during the six-year project cycle of STATCAP-CERDAS. However, one key team member, who was one of the co-ttls at project cancellation, remained with the project since its inception, maintaining project continuity and preserving institutional memory. All the TTLs were based in-country and provided close supervision and implementation support. There was conscious effort to incorporate lessons learned from other public sector reform projects in the portfolio, such as the Government Financial Management and Revenue Administration Project (GFMRAP and the Project for Indonesian Tax Administration Reform (PINTAR. However, STATCAP-CERDAS encountered similar obstacles that could not be overcome. 45. A total of nine Implementation Status and Results Reports were filed during the life of STATCAP-CERDAS averaging almost twice a year, especially after restructuring. The task team had been proactive throughout the project life cycle to document progress and raise issues and challenges on time for management attention. Support missions were fielded to ensure timely resolution of problems encountered, and trust fund resources enabled the provision of complementary TA. However, the task team had not been agile in navigating the multiple leadership changes at BPS, resulting in delays in project implementation at each of these transitions. STATCAP-CERDAS remained a problem project for a prolonged period. 46. To improve BPS implementation capacity and accelerate implementation progress, the World Bank team provided enhanced implementation support throughout the project, from facilitating BPS in the preparation of TOR to selection processes including contract management of various consultancies. The World Bank team also provided procurement training to BPS teams, including training to the Procurement Committees for each selection process. During the selection process for the statistical IT package, when BPS was faced with the challenge of drawing the short Page 27 of 72

28 list of qualified and capable firms for the complex information system package, the World Bank team provided comprehensive guidance and proposed solutions to BPS and proactively worked with BPS and its consultants to conduct the procurement process in an expedient manner. 47. The World Bank could have considered other remedial actions under the prevailing circumstances, given that the grounds for BPS s proposed cancellation of the procurement process of the integrated statistical information systems package were inconsistent with the World Bank s Consultant Guidelines. The World Bank, however, decided not to pursue any further remedial action related to the procurement process, considering that the project in any case was heading for cancellation due to BPS s inability to meet the conditions for lifting suspension. Factors Outside the Control of the Government 48. Indonesia enjoyed relatively stable macroeconomics, and there was no conflict nor natural disaster during the life of the STATCAP-CERDAS project. C. MAIN EVENTS/FACTORS LEADING TO CANCELLATION 49. After restructuring in April 2014, three consultancies provided key inputs to prepare for the design and procurement of the integrated statistical information systems package, which accounted for over 50 percent of the restructured loan amount. These were the Owner s Agent, Statistical Transformation and Quality Improvement TA consultant, and IT Strategy and Design TA consultant. With respect to the information systems package, the Statistical and IT consultants were to provide inputs on the technical requirements for the TOR, while the Owner s Agent consultant was to support the preparation of the Request for Proposal (RFP document. BPS eventually engaged an individual procurement consultant to support the systems procurement, because the Owner s Agent failed to complement its technical specialists with a procurement specialist. 50. The World Bank also conducted implementation support missions in May, August, and September 2016, specifically to support the acceleration of the information systems procurement and keep the project on critical path. These missions provided both technical support for the BPS IT team and training for the procurement and evaluation committees. Training was also conducted in stages to focus on the particular phase of the procurement process, that is, overview training at the beginning of the procurement and one focused on technical evaluation before the RFP closing date. 51. BPS initiated the selection process for the information systems package by issuing the REOI in February 2016, while there were delays in preparing the TOR as described in paragraph 43. After determining only two firms to be qualified for short-listing, BPS proposed to readvertise the REOI to solicit wider interest from qualified firms, and the REOI was readvertised in April 2016, this time also providing more details of the assignment. By June 2016, BPS prepared an initial short list of firms, but upon further clarification of the evaluation criteria, most of those firms were found to be unqualified for this large and complex assignment. BPS sought the World Bank s guidance as advertising the REOI for the third time would have resulted in further delay at this critical stage, while providing no assurance of a different outcome. The World Bank team provided comprehensive guidance and solution to BPS, and in August 2016, BPS completed the revised short list of six qualified international IT firms, which was provided for no objection by the World Bank. BPS proceeded with issuing the RFP in August 2016, and proposals from three firms were received in early October BPS completed the technical evaluation by late October 2016, within four weeks of receipt of proposals, which was a noteworthy performance for an evaluation of such complex nature. However, the Technical Evaluation Report was never sent to the World Bank, pending BPS management approval. The new Chief Statistician requested the Government s internal audit agency (BPKP to review the selection process carried out by BPS so far. Page 28 of 72

29 52. Around this time, BPS s aversion to risks around procurement appeared to be heightened, especially given the high complexity and large value of the package. BPS cited concerns over findings by the Supreme Audit Agency (Badan Pemeriksa Keuangan/BPK for project audit covering , which considered previous selection processes to have procedural deficiencies, even though the World Bank had prior reviewed and issued no objections. Around this time, there was also an ongoing court case involving a BPS official over procurement unrelated to STATCAP- CERDAS. In addition to the new Chief Statistician taking office in September 2016, new officials were appointed to key senior and line manager positions responsible for IT (Deputy Chief Statistician and Director for IT, which also contributed to the break in institutional knowledge along the chain of command regarding the technical discussions and decisions taken with respect to the systems procurement before their appointments. 53. In October 2016, BPKP provided its observations on the procurement process, which included (a issues on procedural compliance that the procurement committee had not obtained prior administrative approval of the cost estimate from the competent authority in the Government before initiating the selection process and the short-listing process was not in compliance with the World Bank s Consultant Guidelines and (b issues on technical aspects such as adequacy of the cost estimate, sufficiency of time provided for proposal submission and for contract execution, and readiness of the TOR. Furthermore, even though the World Bank had prior reviewed and provided no objection to BPS s revised short list of six firms, BPKP provided its own interpretation that the short-listing process was not in compliance with the World Bank s Consultant Guidelines. Specifically, BPKP questioned the inclusion of a third firm from the same country (instead of the maximum two from any one country to meet the short list requirement of six firms, but in BPKP s opinion the short list should have been reduced to five firms while the sixth and only remaining qualified firm should have been excluded from the short list. While the World Bank team clarified the above aspects, including explanation of the principles and intent of the World Bank s Consultant Guidelines, and proposed possible actions to address the technical issues raised in the BPKP review, BPS proposed to cancel the selection process and to start a new selection process. 54. The World Bank did not agree with BPS s proposal to restart the procurement process. After two months of impasse, the World Bank suspended the project in December 2016 with the following conditions for lifting the suspension: (a signing the contract for the information systems package and (b establishment of appropriate counterpart project team for its implementation. During the suspension period, BPS proposed either to relaunch the integrated statistical information system procurement process including updating the TOR and cost estimates and extending the project for an additional two years to close at the end of 2020 or to partially cancel the amount allocated to the package and extend the project for 2.5 years to allow for a 50-month in-house development of an integrated statistical information system financed through loan reallocations. The World Bank did not agree with either of the proposed options, although the team did explore other options with BPS, such as partial cancellation to allow the completion of selected activities under the HR and organization components. 55. Ministry of National Development Planning (Badan Perencanaan Pembangunan Nasional/Bappenas and the MOF also tried to convene meetings with the National Public Procurement Agency (Lembaga Kebijakan Pengadaan Barang Jasa Pemeintah/LKPP and BPKP in an effort to resolve the impasse but were unsuccessful in reaching an agreed way forward. Eventually, with no action from BPS to meet the conditions for lifting the suspension for three months, the World Bank initiated the project cancellation process. STATCAP-CERDAS was formally closed on May 31, Page 29 of 72

30 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E M&E Rating: Modest M&E Design 56. The M&E design did not capture the full picture of the STATCAP-CERDAS implementation progress. The core of the M&E design was the Results Framework with original indicators and baselines collected in annex 1 of the Project Appraisal Document. Achievements at project cancellation were captured in annexes 1 and 7 of this ICR, as well as summarized in Table 2. The PDO was complex, compounded with multiple sub-objectives that could be unpacked in various permutations. This was further complicated by the achievements of the PDO-level outcome indicators, with some exceeding the targets, and the mostly underachieved intermediate results indicators. Of the 17 indicators, 8 were at the PDO level. These PDO indicators provided snapshot readings that did not necessarily capture sustained impacts of the project, compared to those for project-specific intermediate results. The measurements chosen for the PDO indicators were also influenced by improvement efforts made by BPS, as well as through TA from the World Bank and other development partners, notably the ABS. As noted earlier, this was especially the case because the STATCAP- CERDAS framework was initially used to harmonize external technical support. Therefore, some PDO indicators showed statistical quality improvements, despite the implementation lag and low disbursement of STATCAP-CERDAS project. M&E Implementation 57. BPS had conducted surveys and studies to qualify and quantify achievements against the indicators in the Results Framework. All data were collected by the PMU. However, there was no sufficiently robust verification and validation process for the status reported by the subject matter areas. Changes in measurement methodology for user satisfaction surveys, which were not under the World Bank s control, could mean the achieved values may not be directly comparable with the baseline. BPS also made changes to the methodologies for GDP and GDRP compilation, which resulted in corrections to the baseline value for discrepancy. M&E Utilization 58. The World Bank was proactive in monitoring implementation progress, and midcourse adjustments were introduced based mainly on the day-to-day interactions between members of the World Bank s task team and their respective BPS counterparts. Justification of Overall Rating of Quality of M&E 59. Given the significant shortcomings in the M&E system, the overall rating of the quality of M&E is Modest. Page 30 of 72

31 B. ENVIRONMENTAL SAFEGUARD, SOCIAL SAFEGUARD, AND FIDUCIARY COMPLIANCE 60. STATCAP-CERDAS was rated as an environmental category C with no social or environmental safeguards policies triggered. 61. The STATCAP-CERDAS project aimed to improve the statistics essential to support evidence-based decision making in determining government social policies and expenditures for the poor and vulnerable in such areas as health, education, social assistance, and protection. Improved gender-disaggregated statistics would also have supported the Government s efforts for gender mainstreaming in planning and budgeting, as well as M&E of policy and program impacts by gender. 62. BPS was the implementing agency for STATCAP-CERDAS. Both project procurement and financial management capacity assessments were completed at appraisal. The project complied with fiduciary covenants during implementation with internal control arrangements in place. C. BANK PERFORMANCE Overall Rating: Moderately Satisfactory Quality at Entry 63. STATCAP-CERDAS was a carefully designed project that took into consideration international practices and was fully aligned with BPS s internal multiyear plan and reform priorities. Following lessons learned from global STATCAP experiences, the team worked with the client to start by taking stock of its statistical activities and to prioritize the actions to strengthen statistical capacity in the form of its NSDS and/or its statistical master plan. At appraisal, BPS had developed a strategic plan to improve product lines as well as ICT and HR areas. All Echelon 1 and 2 officials of BPS signed off on the strategic plan to signal their commitment. Even though a full-time PMU structure was established during the preparation phase, it was housed in a temporary structure that subsequently had to be dissolved within six months of effectiveness due to changes in bureaucracy reform rules. 64. The design of STATCAP-CERDAS also considered experiences of donor assistance to BPS but overestimated implementation capacity. Past donor assistance to BPS, including the World Bank s, focused primarily on specific technical issues without dealing with organizational issues. Although this approach was successful to a degree, it limited sustained impacts. To avoid this outcome, STATCAP-CERDAS was designed focusing on structural reform that included elements such as ICT governance, HR policies, performance management, and organizational culture. However, the inclusion of three significant information systems implementations in the original scope was too ambitious in light of the limited implementation capacity and may have been better if they had been sequenced. 65. STATCAP-CERDAS was designed in consultation with other development partners to ensure synergies. The preparation of the project involved development partners, such as the ABS, AIPEG, Statistics Netherlands, USDA-NASS, and Statistics Korea. In addition, the MOU between the ABS and BPS for on technical cooperation between the two organizations was deliberately aligned to the STATCAP-CERDAS objectives and areas of support. 66. STATCAP-CERDAS adopted international good practice for institutional reform projects by embedding change management from the design stage. The project was also designed through a broad-based consultative process with both internal and external stakeholders of BPS, intended to foster shared vision and ownership at the inception. Realizing at the outset that institutional reform was a long-term objective, built into the project were opportunities for short-term successes essential for gaining credibility and maintaining momentum of the reform Page 31 of 72

32 processes. In addition, STATCAP-CERDAS was designed for BPS to support the achievement of the Government s ongoing bureaucratic reform initiative. Accordingly, training and development of new skills to adopt the new business processes and use of new ICT applications were factored into the project components. However, the project did not safeguard staff turnover common to multiyear projects. As noted above, with the leadership changes at inopportune junctures of project implementation, STATCAP-CERDAS ultimately suffered from lack of organizational ownership resulting in implementation delays. 67. The project frontloaded detailed analysis and technical requirement specifications during project preparation, but the long preparation period resulted in loss of reform momentum. The long preparation period allowed the project preparation consultant to conduct detailed analysis and prepare the bidding documents ahead of project effectiveness. The use of the PHRD project preparation grant also prolonged the preparation period, because the recipient-executed procurement process for the project preparation consultant was lengthy, and the contract was extended several times to complete the deliverables. By project effectiveness, it had been almost three years since the conception of the project at BPS, and the initial project champion and initiator, the first Chief Statistician, was promoted to a new post in another ministry. 68. Given that official statistics and statistical information systems are highly specialized technical domains, the World Bank could have been more proactive in identifying and providing information on consultants successfully used in other similar World Bank projects both during project preparation and supervision phases. Based on the lesson learned from the weak performance of the project preparation consultant, the statistical and IT systems consultancies were split into two packages. However, this was insufficient to attract qualified consultants, because BPS received weak responses for procurement of key consultancy packages under STATCAP-CERDAS. Both the statistical quality TA and statistical IT consultancy packages required two rounds of REOIs to constitute a short list of qualified consultants, and only two proposals were received out of four short-listed firms for the former, and three proposals out of six short-listed for the latter. Many NSOs, even those in advanced economies, have traditionally developed their own systems, so there are limited commercial-off-the-shelf software specific to statistical infrastructure and few experienced contractors. Quality at Supervision 69. STATCAP-CERDAS was well resourced with country-based TTLs throughout its life, supplemented by subject matter expertise as needed. Subject matter expertise included statistics, IT, institutional/organizational reform, and change management. The task team also proactively provided advice and organized training to strengthen implementation capacity, particularly on project management, procurement, and financial management. TTLs also proactively engaged relevant stakeholders from other government institutions, such as Bappenas, the MOF, and KemenPAN-RB, in efforts to keep them informed and obtain their support. 70. Trust fund resources enabled the provision of complementary TA activities to provide small but concrete demonstrations of statistical integration or technology implementation in practice. These included support for: (a developing the BPS Microdata Catalog System to standardize documentation and publication of survey microdata metadata; (b piloting digital data collection methods through the use of CAPI platform, with Survey Solutions software developed by the World Bank; and (c facilitating peer learning from similar modernization experiences, such as the Turkish Statistics Office. These contributed to some modest improvements but were ultimately not sufficient to sustain reform momentum. 71. The World Bank was proactive in trying to find solutions to address implementation issues, in particular those affecting procurement and financial management processes. Throughout the project, the World Bank team provided proactive procurement and financial management support to the implementation teams. BPS successfully Page 32 of 72

33 completed procurement of four consultancy packages, and even the procurement of the highly complex information systems package was conducted remarkably quickly until the technical evaluation stage. The task team also provided close support to improve FM capacity. Finalization of clear guidelines on payment verification process, clarity of roles within the PMU, and close support from the task team helped the project to improve the timeliness of payment toward the end of the project period. In addition to the extensive hands-on procurement training provided by the World Bank to BPS before and during the selection process of the statistical information systems package, the World Bank team also provided procurement training to more than 200 auditors from BPKP as part of the capacity building of the auditors for their better understanding of the World Bank s Procurement and Consultants Guidelines. However, the BPKP team, which reviewed the ongoing selection process at the request of BPS s leadership, was different from the ones that had been provided procurement training by the World Bank. 72. Given similar experience with the auditors (BPKP or BPK role in other projects and consequently failed procurements, the Government authorities and the World Bank could have done more to anticipate potential issues and provide clarity for implementation agencies. Particularly for complex information systems development, providing procurement training to implementing agencies is necessary but may not be sufficient, as also noted in the Note on Cancelled Operations (NCO for PINTAR. Justification of Overall Rating of Bank Performance 73. Based on the proactive implementation support and TA, as described earlier, the World Bank Performance overall is rated Moderately Satisfactory. D. RISK TO DEVELOPMENT OUTCOME 74. Even though the project was not successful in deploying the IT system for the production and dissemination of statistics, it successfully financed key deliverables that lay the foundations for the development of the IT system with government funding. Among others, the four consultancies financed by the project supported BPS to produce the Statistical Business Framework and Architecture (SBFA with new business processes, the IT strategy, the EA, and the detailed design documents of CSIs. As of the time of the ICR, BPS had secured state funding from the MOF over the next four years to continue its transformation program and implement the statistics information systems development. V. LESSONS AND RECOMMENDATIONS 75. Sequencing long-term reform. Recognizing that public sector reform projects are long-term games, ensuring a phased approach could deliver visible incremental results throughout the project to sustain reform momentum. In this case, a series of short-term goals or quick-wins should be included in the project design, rather than rely solely on a linear, sequential waterfall approach, which may only produce visible results toward the end of a long reform process. The phased approach will demonstrate incremental results and benefits and serve to build implementation confidence early on in the project. When working with complex reform processes, teams should also speed up project preparation to avoid losing reform momentum early on. The new Multiphase Programmatic Approach may be a suitable instrument to support this type of initiative, because it aims to provide more opportunities for adaptation than a single long/large operation, as well as greater continuity than a series of stand-alone operations. 76. Sustaining reform ownership. Committed and visible leadership is key to successful reform, and teams should expect and plan for changes in government leadership in long-term reform processes. When supporting long-term reforms, changes in government leadership can hamper the project outcomes by delaying implementation and undermining ownership in the transition. To manage this risk, it is advisable to both (a build project ownership by engaging and involving staff at all levels of the organization and (b identify and engage external champions that can Page 33 of 72

34 increase mandate and support for reform to drive implementation forward. Embedding change management is critical in institutional transformation initiatives to ease angst during reform implementation and foster buy-in and ownership at all levels. In addition to strong champions within the organization, systematically identifying and engaging the key stakeholders outside the organization who will benefit from and have strong demand for successful transformation will be necessary to gain and sustain political support for the reform continuity. 77. Organizing reform delivery. To effectively deliver institutional transformation, dedicated and empowered teams are required to be isolated from the business-as-usual. Managing institutional transformation programs requires full-time, dedicated teams similar to delivery units, which report directly to the organization s leadership and have the ability to coordinate across the whole organization. Agencies should have the flexibility to design their organization to meet this requirement. Then, appropriate performance incentives should be put in place to attract and secure the best qualified staff. These could range from career or capacity-building opportunities to financial incentives to recognize and reward performance of staff working on reform projects. 78. Planning for complex ICT procurement. Ensure that the prerequisite preparation requirements and enabling conditions are in place before starting the procurement process. ICT transformation is often a key component of public sector reform projects, but procurement of complex ICT such as the development of complex software systems has a high failure rate. Lessons and recommendations from previous operations include an observation from PINTAR that ICT procurement is different requiring both specialized knowledge in ICT and competency in procurement. For projects with ICT-enabled management changes, the World Bank team should give greater attention to ensuring technical and institutional readiness by the project implementing agency. Considering many agencies do not possess sufficient capacity in managing the technical and commercial aspects of large and complex IT system procurement, the World Bank should also explore with the implementing agency on the possibility of drawing the required technical and procurement expertise from specialized departments to supplement the procurement committee, in addition to the external consultant support. In case such expertise is not yet well developed, the Government can also consider developing this function centrally within the competent technical agency, with a view to providing TA to any agency planning procurement and development of complex business or back-office systems. 79. Reorienting audit focus on project performance. The World Bank should work with the Government through training to reorient the auditors focus from strict procedural compliance toward achieving project performance and development outcomes. The focus of auditors on strict procedural compliance has undermined project outcomes of multiple World Bank operations by delaying or blocking complex, especially ICT-related, procurement processes. GFMRAP recommended that the World Bank should work proactively with the Government and the auditors through training to help understand the full scope of the World Bank s Procurement Guidelines to avoid differing interpretations. The World Bank should also explore whether the provisions of loan agreement can be strengthened to make it explicit that the World Bank s decision on the interpretation of the World Bank s Procurement/Consultant Guidelines will be final and binding. 80. Targeting niche markets. Capacity-building projects for specialized agencies, such as NSOs, face common challenges such as weak responses to REOIs and weak consultants, caused by insufficiently developed markets for boutique technical skills. Implementing agencies should conduct active scans of relevant markets to identify the best consultants, individuals, and firms and proactively inform them of business opportunities under the project. World Bank supervision teams can also help by providing information on consultants successfully used in other similar World Bank projects.. Page 34 of 72

35 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Increased effectiveness of the Indonesian National Statistics Office (BPS-Statistics Indonesia to produce and disseminate quality statistics Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Accuracy. Demonstrable improvement in the accuracy of key statistics as evidenced by: (a an increase in the response rate of the Annual Survey of large and medium manufacturing businesses Percentage Apr Dec Dec Dec-2016 Comments (achievements against targets: Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Accuracy. Demonstrable improvement in the accuracy of key statistics, as evidenced by: (b an increase in the user satisfaction with the accuracy Percentage Apr Dec Dec Dec-2016 Page 35 of 72

36 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P of BPS statistics Comments (achievements against targets: Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Relevance. Availability of important new statistics as evidenced by establishment of a national producer price index Text PPI available for agricultural sector, excluding forestry subsector PPI available for all provinces and most large districts PPI available for all provinces and districts National PPI published for the following additional service sectors: Accommodations, Food and Beverage, Passenger Transportation, Electricity and Gas Supply, and Water Supply. Study conducted for the following additional sectors: Freight Transportation, Telecommunications, Financial Services, Education, Health and Waste Management. Preparation for Provincial PPI has been conducted with producer prices survey data collection for Page 36 of 72

37 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P goods and services sectors expanded to all provinces in 2016, and piloting of Provincial PPI for 5 provinces is in progress. 07-Apr Dec Dec Dec-2016 Comments (achievements against targets: National PPI sector coverage has expanded, and data collection has expanded to all provinces to enable Provincial PPI calculation. Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Coherence. An improvement in the coherence of BPS statistics as evidenced by a reduction in the discrepancy between the sum of the Provincial estimates of GDRP and the GDP for Indonesia Percentage Apr Dec Dec Dec-2016 Comments (achievements against targets: Baseline was revised from 27% by BPS, as discrepancy was made 0 (zero in 2010 Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Interpretability. An Text No statistics with fully Metadata available for Metadata available for 310 datasets with Page 37 of 72

38 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P improvement in the interpretability of BPS statistics as evidenced by an increase in the statistics produced by BPS that have supporting meta data documented metadata National Accounts National Accounts metadata information published in BPS Microdata Catalog System 07-Apr Dec Dec Dec-2016 Comments (achievements against targets: BPS makes metadata accessible through the Microdata Catalog System (MCS and Statistics Referral System (SiRuSa Objective/Outcome: Increased efficiency of the Indonesian National Statistics Office (BPS-Statistics Indonesia in producing and disseminating quality statistics Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Timeliness. Improvements in the timeliness of key statistics as evidenced by reducing the time lag in the publication of annual labor force estimates Number Apr Dec Dec Dec-2016 Comments (achievements against targets: Timelag reduced to 90 days in line with timeline for GDP publication Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Accessibility. An improvement in the accessibility of BPS statistics as evidenced by: (a a Number Apr Dec Dec Dec-2016 Page 38 of 72

39 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P dynamic up-to-date web site Comments (achievements against targets: Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Accessibility. An improvement in the accessibility of BPS statistics as evidenced by: (b an increase in the user satisfaction with the accessibility of BPS statistics Percentage Apr Dec Dec Dec-2016 Comments (achievements against targets: A.2 Intermediate Results Indicators Component: Component A: Improved Statistical Quality Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion The successful implementation of a Quality Assurance Framework across BPS Text No corporate quality assurance policy or program Quality assurance program established and effective Quality assurance program established and effective Quality governance structure (unit has been discussed and may be established in 2017; study of quality gates conducted for Page 39 of 72

40 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P Large and Medium Manufacturing Industries Statistics. 07-Apr Dec Dec Dec-2016 Comments (achievements against targets: Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion The establishment of a Common Business Register (BR and a Large Business Unit (LBU Text No BR or LBU available Review of effectiveness of BR maintenance strategy Review of effectiveness of BR maintenance strategy Detailed Design Document for the Statistical Business Register is in finalization phase. 07-Apr Dec Dec Dec-2016 Comments (achievements against targets: Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Availability of an integrated data and information service system supported by the data warehouse Text No integrated data & information system supported by data warehouse available Integrated data & information system supported by data warehouse available Integrated data & information system supported by data warehouse available Statistical Business Framework and Architecture (SBFA, IT Strategy and Enterprise Architecture (EA have Page 40 of 72

41 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P been finalized and endorsed by BPS Executives. Business and functional requirements for Corporate Statistical Infrastructure (CSIs developed and detailed specifications documents are in finalization phase. 07-Apr Dec Dec Dec-2016 Comments (achievements against targets: Component: Component B: Improved ICT Platform and Statistical Information Management Systems Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Improvement in ICT Governance and Management Text ICT Strategy in BPS Strategic Plan ; Existing ICT functions in the Regulation of the Head of BPS No.1/2009 on BPS Organization; Initial business process mapping and analysis All standards available and routinely used as part of "business as usual" Improvement of ICT Strategy, and ICT Governance & Management Framework based on the initial experience and feedback from internal stakeholders BPS IT Strategy, IT Governance and Management Framework / Policy, IT Service Management (Policies, Standards and Procedures, and IT Transformation Roadmap have been finalized and endorsed Page 41 of 72

42 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P in BPS Analysis Document by BPS Executives. 07-Apr Dec Dec Dec-2016 Comments (achievements against targets: Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Development of application systems for statistical production and dissemination Text Systems for common statistical infrastructure do not exist; Statistical processing systems developed by SIS and subject matter areas available N/A - not an indicator originally Application systems for all statistical infrastructure available and in use Business and functional requirements for Corporate Statistical Infrastructure (CSIs developed based on agreed BPS Statistical Business Framework and Architecture (SBFA. Detailed specifications documents for CSIs in finalization phase. 07-Apr Dec Nov Dec-2016 Comments (achievements against targets: Component: Component C: Improved Human Resources Management and Development Page 42 of 72

43 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion A Human Resource that is aligned with BPS's future strategic direction Text No HR Strategic Plan is available; No Technical and Non- Technical Competency Framework and Dictionary are available A workforce that is aligned with BPS's strategic direction Improvement of BPS HR Strategic Plan based on the initial experience and feedback from internal stakeholders; Implementati on of improved HR Strategic Plan Review of current BPS capability, tools, and regulations (high level towards the new business process (SBFA completed, including core & managerial competencies, workload analysis, regulation review, training & development strategy and readiness for new business process. Draft required core & managerial competencies for future BPS are completed, but not yet endorsed by the management. 07-Apr Dec Dec Dec-2016 Comments (achievements against targets: Indicator Name Unit of Measure Baseline Original Target Formally Revised Actual Achieved at Page 43 of 72

44 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P Target Completion An effective, integrated performance management system for BPS Text A performance system that is not in line with BPS's future strategic direction and does not provide sufficient incentive for improved performance An effective performance management system for BPS An effective Performance Management System for BPS BPS-wide Strategy Map completed. Key Performance Indicators using Balanced Scorecard for Echelon I and II are completed, but not yet endorsed by the management. 07-Apr Dec Dec Dec-2016 Comments (achievements against targets: Component: Component D: Improved Organizational Alignment Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Improved alignment of the BPS organization and the future work environment Text BPS organization alignment is not yet optimal An organization design that is well aligned with the BPS's strategic direction Improvement of the organizational design based on the initial experience and feedback from internal stakeholders Organization Structure at macro level (echelon I and II based on GSBPM is completed, but not yet endorsed by the management. 07-Apr Dec Dec Dec-2016 Page 44 of 72

45 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P Comments (achievements against targets: Component: Component E: Project Management Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Effective project management Text No corporately agreed project management system Review of the application of the BPS Project Management System Review of the application of the BPS Project Management System 227 BPS managers have been trained in project management 07-Apr Dec Dec Dec-2016 Comments (achievements against targets: B. ORGANIZATION OF THE ASSESSMENT OF THE PDO Objective/Outcome 1. Increased effectiveness of BPS in producing and disseminating quality statistics Outcome Indicators 1. Accuracy. Demonstrable improvement in the accuracy of key statistics as evidenced by (a an increase in the response rate of the Annual Survey of large and medium manufacturing businesses 2. Accuracy. Demonstrable improvement in the accuracy of key statistics, as evidenced by (b an increase in the user satisfaction with the accuracy of BPS statistics 3. Relevance. Availability of important new statistics as evidenced by establishment of a national producer price index 4. Coherence. An improvement in the coherence of BPS statistics as evidenced by a reduction in the discrepancy between the sum of the Provincial estimates of GDRP and the GDP for Indonesia Page 45 of 72

46 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P Interpretability. An improvement in the interpretability of BPS statistics as evidenced by an increase in the statistics produced by BPS that have supporting meta data Intermediate Results Indicators Key Outputs by Component (linked to the achievement of the Objective/Outcome 1 1. The successful implementation of a Quality Assurance Framework across BPS 2. The establishment of a Common Business Register (BR and a Large Business Unit (LBU 3. Availability of an integrated data and information service system supported by the data warehouse 4. A Human Resource that is aligned with BPS's future strategic direction 5. An effective, integrated performance management system for BPS Component A 1. BPS Statistical Business Framework and Architecture (SBFA and Implementation Roadmap 2. Detailed Design Documents for Corporate Statistical Infrastructure (CSIs (for example, data collection, processing, analysis and dissemination systems; metadata management system; frame and register systems; database and data warehouse 3. Methodological advice on sampling methodologies for continuous household surveys and for pilot of Integrated Horticulture and Estate Crops Survey 4. Establishment of BPS Microdata Catalog System for documentation and publication of survey metadata ( 5. Study visits to NSOs on institutional transformation and statistical integration (Australia, New Zealand, Canada, Turkey, Italy, UNECE 6. Technical training (for example, Short course on Sampling Methodologies for Survey Statisticians in University of Michigan, Agriculture statistics training by USDA-NASS, Peer learning workshop on GSBPM implementation by the Turkish Statistical Institute Component C 7. Gap Analysis Report on Performance Management, Individual and Team Capability, Reward and Recognition, HR Functions, and Training and Development 8. BSC-based Performance Management System for Corporate (until Echelon II Level 9. Job Family Design 10. Core and Managerial Competency Model 11. Individual Performance Management Policy, covering Performance Planning, Implementation and Evaluation 12. Training and Development Strategy and Plan 13. Technical Training (for example, Training Needs Analysis, Assessors, Balanced Scorecard Page 46 of 72

47 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P Change Management 14. Project Launch Workshop in January Change Visioning Workshop in February Planning of Change Champion Kick-Off in March Change Management Report 2015 and Change Progress Online Census (CPOC - Report 2015 and Change Management Program and Work Plan (Quarterly and Induction workshops for BPS staff joining STATCAP-CERDAS PMO, Working Groups and Task Forces in Change Agent Network and Training with identified Change Champions established in HQ and regional offices in June Training on Change Management in HQ and regional offices ( BPS Leaders Session in March Change Management Plan and Strategy in September 2016 Objective/Outcome 2. Increased efficiency of BPS in producing and disseminating quality statistics Outcome Indicators Intermediate Results Indicators Key Outputs by Component (linked to the achievement of the Objective/Outcome 2 1. Timeliness. Improvements in the timeliness of key statistics as evidenced by reducing the time lag in the publication of annual labor force estimates 2. Accessibility. An improvement in the accessibility of BPS statistics as evidenced by (a a dynamic up-to-date website 3. Accessibility. An improvement in the accessibility of BPS statistics as evidenced by (b an increase in the user satisfaction with the accessibility of BPS statistics 1. Improvement in ICT Governance and Management 2. Development of application systems for statistical production and dissemination 3. Improved alignment of the BPS organization and the future work environment 4. Effective project management Component A 1. SAKERNAS CAPI pilot using Survey Solutions; and workshop on lessons learned from CAPI pilots in BPS 2. Technical training (for example, CAPI-Survey Solutions training Component B 3. BPS IT Strategy 4. BPS Enterprise Architecture Page 47 of 72

48 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P IT Governance and Services Management Policies and SOPs 6. BPS IT Transformation Roadmap 7. Technical Training (for example, Enterprise Architecture [TOGAF, RSA]; IT Governance [COBIT]; IT Service management [ITIL]; IT Project Management and Software Development [Agile]; Data Management Component D 8. Gap Analysis Report on Organization and Job Design 9. High-Level Work Load Analysis 10. High-Level Organization Design (including Regional Offices Component E 11. Establishment of project website as socialization platform ( 12. Development of Collaboration Tool Platform 13. Training for PMO and project staff (for example, Training in Organizational Project Management and PMO for Public Organizations, Rapid Results Initiatives Coaching Workshop, Project Management and MS Project, Monitoring and Evaluation, Risk management, Business English Courses, Procurement, Financial Management Page 48 of 72

49 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Preparation Vivi Alatas Budi Permana Christina I. Donna Melinda Good Vivianti Rambe Amien Sunaryadi Dayu Amurwanti Shubham Chaudhuri William Wallace Yoichiro Ishihara Haeduck Lee Noriko Toyoda Roksana Khan Erwin Ariadharma Role TTL Procurement Specialist Financial Management Specialist Country Lawyer Environmental Safeguards Specialist Governance and Anticorruption Specialist Governance and Anticorruption Specialist Sector Manager Team Member Team Member Team Member Team Member Team Member Team Member Supervision/ICR Noriko Toyoda, Erwin Ariadharma TTLs C. Bernard Myers TTL Theo David Thomas TTL Achmad Zacky Wasaraka Procurement Specialist Budi Permana Procurement Specialist I Gusti Ngurah Wijaya Kusuma Financial Management Specialist Christina Donna Financial Management Specialist Ahsan Ali Team Member Yash Gupta Team Member Page 49 of 72

50 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P Tina George William S. Hebb Agus Suherman Lina Lo Nicholas Menzies Carlos C. D. Ferreira Team Member Team Member Team Member ICR Author ICR Team Member ICR Team Member B. STAFF TIME AND COST Stage of Project Cycle No. of staff weeks Staff Time and Cost US$ (including travel and consultant costs Preparation FY , FY , FY , FY , FY , FY , FY , FY , FY15 0 3, FY Total , Supervision/ICR FY , FY , FY , FY , FY , FY , Page 50 of 72

51 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P FY , Total , Page 51 of 72

52 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Closing Percentage of Approval Components (US$, millions (US$, millions (% Component A: Improved Statistical Quality Component B: Improved ICT Platform and Statistical Information Management Systems Component C: Improved Human Resources Management and Development (including BPS Staff Training and Capacity Building + Component D: Improved Organizational Alignment Component E: Project Management Total Page 52 of 72

53 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P ANNEX 4. EFFICIENCY ANALYSIS N/A Page 53 of 72

54 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P ANNEX 5. BORROWER S COMPLETION REPORT AND/OR COMMENTS Page 54 of 72

55 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P Page 55 of 72

56 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P Page 56 of 72

57 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P Page 57 of 72

58 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P Page 58 of 72

59 Statistical Capacity Building-Change and Reform for the Development of Statistics (STATCAP-CERDAS ( P Page 59 of 72