Pakistan: Reproductive Health Project

Transcription

1 Validation Report Reference Number: PCV: PAK Project Number: Loan Number: 1900(SF) December 2010 Pakistan: Reproductive Health Project Independent Evaluation Department

2 ABBREVIATIONS ADB Asian Development Bank DHPT district health and population team EmOC emergency obstetric care FWW family welfare worker IEC information, education, and communication IED Independent Evaluation Department LHW lady health worker MOH Ministry of Health MOPW Ministry of Population Welfare MSU mobile service unit NGO nongovernment organization PC-1 Planning Commission Form 1 OCR ordinary capital resources PCR project completion report PCU project coordination unit PIU project implementation unit RHC rural health center RRP report and recommendation of the President TA technical assistance NOTE In this report, $ refers to US dollars. Key Words adb, asian development bank, coordination units, family health, independent evaluation department, mobile service unit, pakistan, project completion report, reproductive health, validation Director R. B. Adhikari, Independent Evaluation Division 1, Independent Evaluation Department (IED) Team leader Team members S. Shrestha, Evaluation Specialist, Independent Evaluation Division 1, IED C. Infantado, Portfolio Evaluation Officer, Independent Evaluation Division 1, IED S. Labayen, Senior Operations Evaluation Assistant, Independent Evaluation Division 1, IED V. Melo, Operations Evaluation Assistant, Independent Evaluation Division 1, IED In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, the Independent Evaluation Department does not intend to make any judgments as to the legal or other status of any territory or area.

3 PROJECT COMPLETION REPORT VALIDATION FORM A. Basic Project Data PCR Validation Date December 2010 Project and Loan Number 30210, 1900(SF) Approved ($ million) Actual ($ million) Project Name Reproductive Health Project Total Project Costs Country Pakistan Loan Sector Health, Nutrition, and Social Total Cofinancing Protection ADB Financing ADF: Borrower OCR: 0.00 Beneficiaries Cofinanciers none Others Approval Date 20 Dec 2001 Effective Date 20 Jul Feb 2004 Signing Date 20 Mar 2003 Closing Date 30 Jun Aug 2009 Project Officers Name: Vincent de Wit Hiroyoshi Kurihara Vincent de Wit Sekhar Bonu Manuel Perlas Yukihiro Shiroishi Raushan Mamatkulov Yukihiro Shiroishi R. Narasimham Location From Nov 2001 Jan 2004 Oct 2004 Dec 2004 Jul 2006 Sep 2006 Dec 2006 Sep 2007 Mar 2008 To Dec 2003 Sep 2004 Nov 2004 Jun 2006 Aug 2006 Nov 2006 Aug 2007 Feb 2008 Dec 2009 Validator: M. Gorra Director: R. B. Adhikari, IED1 Consultant Quality Control Reviewer/Peer Reviewer: S. Shrestha Evaluation Specialist, IED1 ADB = Asian Development Bank, ADF = Asian Development Fund, IED1 = Independent Evaluation Division 1, OCR = ordinary capital resources, PCR = project completion report, SF = special fund. B. Project Description (Summarized from the report and recommendation of the President [RRP]) 1 (i) Rationale. The Government of Pakistan was concerned about the poor health status of women and children and the implications of high population growth for family welfare, the environment, and economic development. It wanted to accelerate the decline in fertility, introduce comprehensive reproductive health services, and promote gender equity and women s empowerment. The government planned a transition from separate family planning and maternal and child services to an integrated reproductive health approach to improve the acceptability, efficiency, and impact of these services. Priority was given to the poor in the least developed and underserved districts. Consequently, the project targeted 34 districts in the third and fourth quartiles in terms of social ranking. At least 50% of targeted project beneficiaries were poor. (ii) Impact. The overall goal of the Reproductive Health Project was to improve the reproductive health status of families, reduce maternal and infant mortality, and reduce population growth, to contribute to improving women s social status and reducing poverty. Specific performance indicators and targets were to: (a) reduce the population growth rate from 2.6% to 2.3% in targeted districts, (b) reduce the total fertility rate from 5.5% to 5.0%, (c) reduce the infant mortality rate from 112 to 90 per 1,000 live births, and (d) increase the contraceptive prevalence rate from 15% to 22%. 1 ADB Report and Recommendation of the President: Proposed Loan to Pakistan for the Reproductive Health Project. Manila.

4 2 (iii) Objectives or Expected Outcomes. The project had a dual objective of (a) increasing the use of reproductive health services in 34 poor and underserved districts, and (b) strengthening the sector capacity for reproductive health services in the context of the government s restructuring process. The project was designed to improve the range and quality of reproductive health services through the development of health human resources and a health-care infrastructure, with parallel efforts to improve client awareness and education, and harness private sector and nongovernment organizations (NGOs) to expand reproductive health services. Specific end-of-project performance indicators and targets were: (a) doubling the use of reproductive health services within 6 years, (b) increasing the use of reproductive health packages to 70% of outlets, (c) doubling access to lady health workers (LHWs), (d) adding one hospital in each district to provide a 24-hour comprehensive emergency medical obstetrics clinic (EmOC), (e) reaching 50% of the targeted information, education, and communication (IEC) audience, (f) approving provincial reproductive health programs for expansion, (g) fully establishing reproductive health directorates, (h) implementing reproductive health services per annual and 5-year plans for all 34 district health and population teams (DHPTs), (i) upgrading all targeted training institutions, and (j) nongovernment organizations (NGOs) and private sector expanding reproductive health services in targeted districts. (iv) Components and/or Outputs. As designed, the project had six components: (a) improve the quality and range of reproductive health services through skills-based, in-service training, provision of essential equipment and supplies, and quality control through supervision and monitoring; (b) improve access of the rural poor to community-based reproductive health services through family welfare workers (FWWs), and selective upgrading of family planning and obstetric care services; (c) increase awareness about reproductive health services and changing practices among specific target groups through the IEC and advocacy among local leaders; (d) strengthen the sector capacity for reproductive health services by aligning national and provincial reproductive health structures and management to support the reproductive health approach, integration of reproductive health services, development of provincial reproductive health programs, and devolution of responsibilities and authorities to the districts; (e) develop human resources for reproductive health services by improving pre-service training capacity; and (f) harness NGOs and private sector capacity with financial support and training to develop reproductive health services. Specific performance indicators and targets were: (a) 13,800 female village-based family planning workers (VBFPWs), 1,300 LHWs, and 300 female and 2,550 male doctors trained; (b) 90% of medicine and contraceptive supplies available; (c) 4,400 LHWs and 1,200 male VBFPWs added in targeted districts; (d) 47 mobile service units (MSUs) operational; (e) surgical contraception provided in 33 new reproductive health centers; (f) 34 hospitals upgraded to provide comprehensive EmOC; (g) a federal communication group reactivated and meeting on a quarterly basis; (h) LHWs and male VBFPWs trained; (i) awareness raising media activities in 34 districts; (j) ward meetings in 34 districts; (k) 20 TV/radio spots and 10 drama serials; (l) full staffing, funding, and logistics; (m) staff training in district management systems in 34 districts; (n) strengthen plans for women s health in 34 district; (o) reproductive health programs prepared by year 2; (p) 2,000 midwives (full course) trained/graduated; (q) increased knowledge and improved skills in reproductive health of 500 FWWs, 4,000 lady health visitors, 15,000 medical students, and 300 postgraduate doctors by imparting RHspecific training; (r) 2,500 private practitioners, 4,000 paramedics, and 8,000 pharmacists trained; (s) 147 FWW supervision vehicles and 28 new vehicles procured, and 19 refurbished for MSU use; and (t) 33 new and 8 old reproductive health centers upgraded to type A, plus reproductive health equipment and supplies for 2,600 basic health units and family service centers. Additionally, the project was to: (a) procure 34 group media vehicles, 11 vehicles, and 5 fellowships for project management and 23 vehicles for public health schools; (b) upgrade 12 midwifery schools and develop a curriculum for 13 Ministry of Population Welfare (MOPW) training institutes; and (c) support fellowships for 120 subdistrict training facilities. A training program for private-sector and NGO providers was also included in the design.

5 3 C. Evaluation of Design and Implementation (Project completion report [PCR] 2 assessment and validation) (i) Relevance of Design and Formulation. During appraisal the project design was deemed relevant as it was consistent with both the poverty reduction strategy and country assistance plan for Pakistan of the Asian Development Bank (ADB) as well as the government s policies and plans. The project s relevance was, however, considerably eroded by what the PCR described as its complex and ambitious implementation structure, with 52 entities involved in day-to-day work management. Moreover, the PCR identified several factors that adversely affected implementation: (a) weak institutional commitment and capacity at both the federal and provincial levels; (b) failure of the two key federal ministries involved in the project the Ministry of Health (MOH) and the MOPW to coordinate; (c) weak coordination between federal and provincial governments; (d) none of the six executing agencies (the MOPW, the MOH, and four provincial health departments) or the four implementing agencies was able to set up a project coordination unit (PCU) and a project implementation unit (PIU) as envisaged in the project s design; and (e) despite the complexity of the project and its coordination arrangements, a project administration manual was not prepared in the early stages to guide ADB and the government in implementation. Increased political impediments and rapidly changing priorities was also noted by the PCR to have seriously diminished the relevance of the project s design. This validation agrees with the PCR that the institutional arrangements for project implementation were complex, considering that in the minutes of the Staff Committee Review meeting, one committee member explicitly cited numerous institutional constraints in Pakistan that raised the issue of project implementation capacity. Why these were overlooked in designing the project s institutional arrangements is therefore surprising, given that assessment of institutional capacity was one of the areas highlighted in the feasibility study s terms of reference. The relative complexity of the design itself may have influenced the institutional arrangements. (ii) Project Outputs. The project did not perform well and did not achieve its targets in all components. Overall, the PCR noted that the project achieved only 30% of its physical targets: only about 10% of target training of various categories of health and/or family planning personnel was actually accomplished, with training quality assessed to be generally low; only three of 34 civil works for upgrading of EmOC services in rural health centers (RHCs) and provincial hospitals were completed, and only six of a targeted 28 MSUs were established; minor repairs were made in only 151 of a targeted 2,500 basic health units; of the targeted 4,400 female FWWs, the MOH recruited only 2,200 LHWs and 88 lady health supervisors, all of which were recruited only in the last 2 years of the project, while the MOPW recruited 494 male mobilizers of 1,200 targeted during appraisal. None of the 33 additional RHCs were constructed; instead, only minor repairs were made in 10 RHCs. (iii) Project Cost, Disbursements, Borrower Contribution, and Conformance to Schedule. As many activities were not carried out and outputs not generated, at loan closing the project had used only $8.44 million (18.8%) of the $45 million estimated at appraisal. ADB financed a total of $5.68 million, comprising about $3.35 million in foreign currency (40%) and about $2.33 million in local currency costs (28%), and the government contributed about $2.76 million (33%). The PCR noted considerable delays in making counterpart funds available to executing and implementing agencies. In some cases, funds were received less than a year before the closing date. The PCR added that while most of the loan proceeds were disbursed through the imprest account, the turnover ratio was very low (0.05 as of 31 December 2008). The first disbursement was made on 28 January 2005 and the final disbursement on 18 August Disbursement was low in the first 3 years of project implementation (6% against an elapsed period of 62%), but was reported to have picked up slightly after all the PCUs and PIUs were established by early 2006, 4 years after project approval, and less than 2 years before the closing date. The project was designed for 6 years, starting in July 2002, but experienced significant delays in implementation. The loan agreement, which was signed on 20 March 2003, set the date of loan 2 ADB Completion Report: Reproductive Health Project in Pakistan. Manila.

6 4 effectiveness as 20 June The loan did not become effective until 24 February 2004, however, more than 2 years after its approval by ADB s Board of Directors (20 December 2001), mainly because of the inability of the executing agencies to establish the PCUs and secure government approval of their respective Planning Commission Form 1 (PC-1). Why this was so was not explained in the PCR, nor in the other project documents, but clearly the establishment of PCUs and approval of PC-1s are important elements in successful project implementation. The government s failure to put these into place may have seriously jeopardized the project s implementation. PCUs are critical to effective project management, and PC-1s were deemed essential for securing counterpart funds. Notwithstanding the delay in the project s start, the closing date of 31 December 2007 was maintained, leaving barely 4 years for actual implementation. It is not clear whether the closing date was not extended because the project was flagged for advance closure under an ADB-government agreed plan for strengthening portfolio performance, wherein prompt corrective actions were to be taken for poorly performing projects. This advance closure, however, did not materialize because of a failure to agree between the government and executing agencies involved. Project documents indicated that at least one executing agency questioned the motion for advance closure and even requested an extension of the closing date, citing the late project start. The PCR stated that the PCUs and PIUs were established 2 years after loan effectiveness, and their staff members were mostly inexperienced and unfamiliar with ADB procedures. Why the government did not recruit more qualified staff sooner was not explained in the PCR and the project documents. Neither was there an explanation of why ADB allowed this to happen, without proper mitigation (i.e., provision of adequate pre-service or on-the-job training). With barely 2 years to catch up with the delays in implementation, it is not surprising that the project achieved only 30% of its physical targets and spent only 18.8% of the estimated total project cost. The PCR added that ADB fielded only one loan review mission in February 2005, a year after the loan was declared effective. This was surprising, considering that the project clearly needed closer supervision, to ensure that (a) conditions for effectiveness and loan covenants were successfully complied with and on time, and (b) to ensure that a proper project management structure was in place to oversee project implementation. Another ADB loan review mission was fielded at the end of July 2006, but could not be completed because the lone mission member had a personal emergency. Neither the PCR nor the project documents reviewed by this validation provided an adequate explanation as to why ADB did not urge the government to speed up the implementation process, adhere to the loan covenants and conditions to loan effectiveness, and establish proper PCUs and PIUs. The PCR noted that none of the PCUs or PIUs was staffed according to the project design, in terms of both numbers and quality. Nowhere in the project documents was there any suggestion that ADB had drawn the government s attention to this apparent anomaly. (iv) Implementation Arrangements, Conditions and Covenants, related Technical Assistance, and Procurement and Consultant Performance. The PCR described the implementation arrangements designed for the project as highly structured, complex, ambitious, and unrealistic, and further noted that compliance with the loan covenants had generally been unsatisfactory. The borrower complied fully with seven of the 19 specific covenants, partly complied with eight, did not comply with three, and one covenant was not applicable. Of the 23 general covenants, the borrower fully complied with 1, partly complied with 11, did not comply with 10, and 1 was not applicable. The covenants requiring joint committees to meet quarterly and the project steering committees to meet at least once a month were deemed unrealistic. Similarly, the covenant pertaining to the composition of PCUs, PIUs, and DHPTs was deemed overly ambitious. This validation considers that the complexity and importance of the project would have required at least a properly staffed PCU and/or PIU at various levels, and that the executing and/or implementing agencies should have demonstrated initiative in recruiting and staffing, and compensated for inadequacies in skills by providing appropriate on-the-job or pre-service training. Not a single PCU or PIU had the number and quality of project staff planned at appraisal and the DHPTs were almost nonexistent. The covenant for benefit monitoring and evaluation was not

7 5 implemented at all. Baseline data were not collected nor were end-of-project surveys conducted to evaluate the project s benefits. Since the actual project start occurred almost midway through the original schedule, no midterm review was undertaken. The PCR seems to have relied mainly on the quarterly project reports and its own limited surveys to measure progress or lack thereof. In the absence of a baseline survey and end-of-project surveys, the PCR made no attempt to measure project benefits. A technical assistance (TA) fact-finding mission, which visited Pakistan from 12 to 17 July 1999, reached an understanding with the government on the objectives, scope, cost estimates, financing plan, implementation arrangements, and consulting services for a project preparatory TA to conduct a feasibility study and prepare a project proposal and implementation plan to help the MOPW and other agencies improve the reproductive health status of families, in particular of the rural poor, and reduce fertility and maternal and infant mortality. The TA, which was completed in December 2000, resulted in the RRP for the Reproductive Health Project, identified key issues hindering the use of reproductive health services, including institutional constraints (i.e., weak financial management and personnel constraints), but the resulting project design addressed mainly technical issues (i.e., inadequate access, poor quality services, etc.). The PCR opined that a simpler implementation arrangement could have been designed, given the identified institutional constraints. Because of the weak capacity of the executing and implementing agencies, a sizeable set of consulting services was included in the design to make sufficient TA available to the project staff. This included about 30 person-months of international consulting services and about 180 person-months of national consulting services, to be contracted through a firm. No firm was, however, recruited and no individual consultant was hired under the project, leaving the inexperienced project staff without adequate management support to navigate the complex implementation arrangement. The government seems to have had little or no confidence in the capacity of consultants to provide useful services to the project, citing previous experience in other projects where consultants were said not to have delivered added value to project management. No clear explanation was given as to whether the government failed to find suitable consultants or whether it simply chose not to hire consultants because of preconceived notions that consultants do not add value to the total project effort. In the end, both ADB and the government should have jointly reviewed this decision and agreed on practical measures to address the government s concerns about consultant quality and usefulness, without compromising the project management s need for TA and expert support. The PCR noted that performances of civil works contractors were reasonably satisfactory in terms of timely completion, but that the quality of works was not up to standards. No explanations were given as to why this was so, but it could either have been due to the industry norm, or to a failure of project monitoring and supervision. Likewise, suppliers of medical equipment, hospital furniture, instructional materials, and medicine were noted to have generally complied with specifications in quality, but did poorly on post-sales service and maintenance. The PCR thus rated the performance of the contractors as partly satisfactory and this validation concurs with the rating, purely on the basis of the PCR s observation on the quality of work and post-sales service. (v) Performance of the Borrower and Executing Agency. The PCR viewed the delayed release of counterpart funds and establishment of PCUs and PIUs as a reflection of the borrower s inadequate commitment to the project. It therefore rated the performance of the borrower and executing agencies unsatisfactory. In view of the foregoing and given the borrower s apparent lack of effort to improve implementation performance and adhere to the covenants and project agreements, this validation concurs with the rating. (vi) Performance of the Asian Development Bank. Despite inordinate delays in borrower compliance with the covenants and numerous problems causing the late start in project implementation, ADB did not seriously call the attention of the executing and/or implementing agencies to the matter and conducted only one review mission during the entire project. Another loan review mission was fielded

8 6 toward the end of July 2006, but the PCR noted that this was not completed because the lone mission member had a personal emergency. Also, the PCR noted that some project staff complained about significant delays in ADB approval of requests for procurement and disbursement. A high turnover of sector divisions and project officers was also noted. Although project implementation appears to have been stymied from its conception, ADB s internal project performance monitoring system did not trigger any warning signs until early 2007, when it was too late to make up for implementation delays. The PCR acknowledged that since mid-2007 staff at headquarters and the resident mission worked closely to support project implementation, but that this did not compensate for the other difficulties faced in the project. Thus, from July 2007 to 18 August 2009, the project performance rating dipped to unsatisfactory for implementation progress, although it was still rated partly satisfactory on development objectives. The PCR rated ADB s overall performance unsatisfactory, and in view of the foregoing, this validation concurs with the PCR rating. D. Evaluation of Performance (PCR assessment and validation) (i) Relevance. The PCR rated the project partly relevant. Although its objectives to help improve the quality of health and family planning services and strengthen reproductive health services were relevant and the project s technical design adequately provided a strategy for reducing maternal and infant mortality and improving women s health, the project seriously erred in assessing the capacity of the executing and implementing agencies to implement this innovative, but highly complex project. The PCR also noted that the project preparatory TA consulted many stakeholders, which might have secured ownership of the project by a broad segment of potential project participants, but which in the view of this validation might have also contributed to the project s complexity by expanding the scope of the project beyond the institutional capacity of the executing and/or implementing agencies. Indeed, the project s technical design covered all the important elements that could influence reproductive health, maternal/child mortality, and gender equity, but managing such a complex design became a challenge in the light of earlier observations about institutional constraints, and the project design failed to ensure adequate implementing capacity. In view of the foregoing, this validation concurs with the PCR rating and deems the project partly relevant. (ii) Effectiveness in Achieving Outcome. The PCR rated the project ineffective. This validation concurs with the PCR rating because by not being able to deliver substantially on the expected outputs, the project may have also failed to achieve its two main objectives: (a) to improve reproductive health services in 34 targeted districts; and (b) to strengthen the sector s capacity for reproductive health services in the context of the government s restructuring process, including through NGOs and the private sector. Unfortunately, no data was collected on project outcomes and benefits because the planned benefit monitoring and evaluation surveys were not done. The level of physical progress in output objectives (30%) achieved in target districts was insufficient to create the critical mass needed to produce the desired improvement in the use of reproductive health services. Furthermore, this validation agrees with the PCR assessment that political exigencies and institutional resistance to change within the MOH and the MOPW could have stymied the government s restructuring agenda for reproductive health services. The initial plans to merge the MOH and the MOPW into one ministry of health and population welfare did not materialize because of competing interests. (iii) Efficiency in Achieving Outcome and Outputs. The PCR rated the project inefficient. The PCR noted that no financial or economic analysis was done at appraisal. Instead, project benefits were estimated using the disability adjusted life year approach. The PCR noted further that the estimated savings of disability adjusted life years was $21, using national averages, and not the district-specific rates of the disease. Not only was this not considered valid for assessing project efficiency, the PCR also considered any calculations at this time as too soon because there are considerable time lags before changes in the maternal mortality ratio, infant mortality rate, fertility rate, and other indicators can be observed. Given what the PCR noted as significant delays caused by inefficient project management, limited use of loan funds earmarked for its different components, unsatisfactory reports of service delivery because of termination of contracts of VBFPWs and FWWs recruited under the

9 7 project, and continued bifurcation of reproductive health services between the MOH and the MOPW, this validation concurs with the PCR rating. It rates the project inefficient on the basis of project management issues that seriously downgraded project efficiency in achieving outcomes and outputs. (iv) Preliminary Assessment of Sustainability. The PCR rated the sustainability of project interventions unlikely and this validation concurs with the rating, given the continued fiscal challenges and financial constraints in public sector financing. Moreover, the scope of reproductive health services has been significantly reduced with the termination of service contracts of the VBFPWs and FWWs, which had been paid from project funds. (v) Impact. The PCR did not assess project impact, but noted that the project may have had a moderate impact on the knowledge and attitudes of target communities and health workers toward reproductive health issues. Whether this translates into good reproductive health practice is yet to be seen. Such knowledge and attitudes are generally poor predictors of behavioral change. Given the absence of baseline and end-of-project survey data, this validation cannot objectively assess the project s impact, but given its limited output and unlikely sustainability, the validation rates the impact of the project as negligible. E. Overall Assessment, Lessons, and Recommendations (Validation of PCR assessment) (i) Overall Assessment. Overall, the PCR rated the project unsuccessful, noting that the project was partly relevant, ineffective, inefficient, and unlikely to be sustained. The PCR further noted that the project was unable to employ sufficient inputs for producing the desired outputs and outcomes. It performed poorly on procurement of goods, services, works, and implementation of soft components such as capacity building and IEC. This validation report agrees with this rating, considering that most of the problems identified are not new and could have been anticipated, or at least mitigated. For example, the project could have provided intermediate TA to help set up and train the PCUs and PIUs, and conducted workshops with key executing and/or implementing agency project officers to map out the preliminary activities and make sure that adequate counterpart funds were available, imprest accounts were opened, and project requirements were included in the PC-1. Previous projects in the sector have experienced similar problems, yet the project design was overly optimistic in putting together a project with a broad range of activities and a complex implementation arrangement, which required the recruitment and training of project staff and program personnel on a level that the PCR deemed unrealistic. Also, having already identified institutional constraints as a likely impediment to project implementation, the project provided for extensive TA, fielding international and domestic consultants to help project staff. When the borrower appeared to have rejected the TA, even as it failed to establish a proper project management team, ADB should have been alarmed and initiated appropriate action to remedy the situation. That ADB did nothing at the time, and fielded its first review mission only a year after loan effectiveness, which in itself was already delayed by over 2 years, is difficult to justify, given the importance of this project as rationalized in the RRP. A second review mission, fielded in July 2006, could not be completed. (ii) Lessons. The PCR highlighted the need for a detailed objective analysis of the potential implementation risks and for necessary mitigation measures to be built into the project design, including a fair assessment of the executing and/or implementing agency s capacity and readiness to implement the project. This was supposed to have been addressed by the TA as part of its feasibility study. This validation, therefore, does not consider this to have been the core problem, as the government fully accepted the TA design. What seems to have been a problem was the government s lack of willingness, capacity, or commitment to comply with the covenants and conditions for loan effectiveness. This brings into question the degree of ownership that the government had in the project. The extensive stakeholder consultations conducted during project preparation should have dealt with the ownership issue. It is commendable that many stakeholders participated in these consultations, but the principal stakeholders whose ownership of the project needed to be secured were the government and its executing and/or implementing agencies. It is also possible that project staff, on both the

10 8 government s and ADB s side, failed to ensure that the conditions needed to meet the loan covenant were properly documented and signed for compliance purposes. For example, ADB s inadequate review of the PC-1 and inconsistencies between the PC-1 and the RRP are perennial problems that always slow the pace of project implementation, as they did for this project. For this validation report, the main lessons learned for implementing projects in countries with similar circumstances as in Pakistan are: (a) the need for start-up TA to set the stage for effective project implementation as part of the loan covenant, establish the PCUs and/or PIUs, and ensure that a realistic implementation arrangement is put in place, even if it means revising the project implementation design, drafting a management operation manual, assisting in the opening of imprest accounts, training project staff on ADB procurement and disbursement procedures and reporting requirements, and initiating the preparation of procurement and disbursement documents; (b) the need for more frequent review missions, probably with an appropriate consultant/advisor on board, in the early years of implementation and for as long as project implementation is deemed off track; and (c) the need to take immediate action to scale down project costs, and/or cancel part of the corresponding loan, when it becomes apparent that the government is unable or unwilling to put up the necessary counterpart funds. While it is important to correctly and objectively assess implementation constraints and implementation capacity, countries with serious development needs and weak institutional capacity should be assisted with building up this capacity as part of project design. This refers to short-term project implementation capacity as well as to longer-term program administration capacity. The project starting date should be set after start-up TA has been successfully completed, and countries with weak institutional capacity should be given time to comply with the covenants and requirements for loan effectiveness through a mentoring process that the start-up TA should provide. (iii) Recommendations. The PCR recommends that future projects secure sustained commitment of funds from the borrower to the executing and/or implementing agencies. Furthermore, it suggests that measures be introduced to strengthen record keeping by executing and/or implementing agencies to facilitate future ADB review missions, and that ADB explore whether its fiscal support operations can target critical health sector financing requirements. This validation further recommends the following: (a) assess borrower countries and corresponding prospective executing and implementing agencies for institutional capacity and readiness to implement development projects based on past project implementation experience, staff qualifications and availability, willingness to recruit additional staff if needed, and the presence of other institutional constraints. Countries with intense sector and development needs but weak institutional capacity should be prioritized for project start-up TA and mentoring support to be completed within 12 months of loan approval. The TA should help set up the PCU and/or PIU and assist in meeting loan covenants and conditions to effectiveness; (b) establish parameters and procedures for determining whether or not a project needs more frequent review missions; and (c) establish parameters and procedures for immediate special action to scale down project costs and corresponding loans whenever a borrower experiences difficulty in putting up the necessary counterpart funds. F. Monitoring and Evaluation Design, Implementation, and Utilization (PCR assessment and validation) Although the project design provided for a midterm review, baseline surveys, and end-of-project surveys, the PCR noted that no monitoring and evaluation system was put in place for the project. No explanations were given, but this validation views this as another illustration of how project management failed to implement activities included in the project design. This may either have been an active decision to selectively implement only those activities for which management was able to secure funding or approval for funding, or it could simply be one of the many project activities that failed to be implemented.

11 9 G. Other (e.g., safeguards, including governance and anticorruption provisions; fiduciary aspects; government assessment of the program/project, as applicable) (PCR assessment and validation) Except for environmental control and safeguard measures, which are part of the covenant requirements, no other issues were highlighted. The PCR noted that the project did not have any significant adverse environmental or sociocultural impacts. Since civil works focused primarily on improving existing facilities, they involved no resettlement in any of the project sites. Independent Evaluation Department H. Ratings Project Completion Report Review Relevance Partly relevant Partly relevant Effectiveness in Ineffective Ineffective Achieving Outcome Efficiency in Achieving Inefficient Inefficient Outcome and Outputs Preliminary Assessment Unlikely Unlikely of Sustainability Borrower and Executing Unsatisfactory Unsatisfactory Agency Performance of Asian Unsatisfactory Unsatisfactory Development Bank Reason for Disagreement/Comments Impact Moderate Negligible The PCR considers that the project may have had a moderate impact on the reproductive health knowledge and attitudes of health workers and the community. This validation, however, notes that knowledge and attitudes are not part of the performance indicators for impact, and are poor predictors of behavioral change. With limited outputs and activities, however, it is doubtful whether any significant improvement in reproductive health impact indicators could have resulted from the project. Overall Assessment Unsuccessful Unsuccessful Quality of Project Completion Report I. Comments on PCR Quality Satisfactory The PCR compiled an extensive amount of data and information on project outputs and implementation activities. It presented a well-researched account of project implementation and was forthright in citing the problems met, although it could have provided clearer explanation of some decisions and actions or inactions made by the borrower and ADB relating to some aspects of implementation. The PCR is well written, candid, and forthright in bringing out the issues that hampered implementation.

12 10 J. Recommendation for Independent Evaluation Department Follow-Up IED should consider closely reviewing all unsuccessful projects like this one to draw lessons for future operations. It may be expedient to simultaneously review a package of projects in countries like Pakistan with a high failure rate. This will provide general insights into both country- and sector-specific institutional constraints, which will be useful in designing future projects and loans. K. Data Sources for Validation The main data sources for this validation are the RRP, the PCR, and several project files dealing with the project preparatory TA, including minutes of Staff Review Committee meetings, project progress reports, and back-to-office reports of the TA fact-finding mission, TA inception mission, final TA review mission, loan fact-finding mission, and loan review mission, among others.

13 REGIONAL DEPARTMENT S RESPONSE TO THE PROJECT COMPLETION REPORT VALIDATION REPORT On 15 November 2010, the Independent Evaluation Department (IED) circulated the draft validation report for interdepartmental comments. IED received helpful editorial comments from the Pakistan Resident Mission of the Central and West Asia Department, and these were duly incorporated into the final draft. The resident mission reviewed the final report, supports the changes made by IED.