United Nations Development Programme. United Nations Development Programme

Transcription

1 United Nations Development Programme United Nations Development Programme Terminal Evaluation of UNDP/GEF Project: Sustainable Urban Transport Project (Component 1A) (GEF Project ID: 3241; UNDP PIMS ID: 3214) Terminal Evaluation Report Mission Members: Mr. Roland Wong, International Consultant Dr. Sudhakar Yedla, National Consultant June 2017

2 UNDP Government of India Terminal Evaluation of Sustainable Urban Transport Project (Comp.1A) TABLE OF CONTENTS SYNOPSIS... III EXECUTIVE SUMMARY... IV ABBREVIATIONS... IX 1. INTRODUCTION... 1 Page 1.1 PURPOSE OF THE EVALUATION SCOPE AND METHODOLOGY STRUCTURE OF THE EVALUATION PROJECT DESCRIPTION AND DEVELOPMENT CONTEXT PROJECT START AND DURATION PROBLEMS THAT COMPONENT 1A OF SUTP SOUGHT TO ADDRESS IMMEDIATE AND DEVELOPMENT OBJECTIVE OF COMPONENT 1A OF SUTP BASELINE INDICATORS ESTABLISHED MAIN STAKEHOLDERS EXPECTED RESULTS FINDINGS PROJECT DESIGN AND FORMULATION Analysis of Project Planning Matrix Risks and Assumptions Lessons from Other Relevant Projects Incorporated into Component 1A Design Planned Stakeholder Participation Replication Approach UNDP Comparative Advantage Linkages between Component 1A of SUTP and Other Interventions within the Sector Management Arrangements PROJECT IMPLEMENTATION Adaptive Management Partnership Arrangements Feedback from M&E Activities Used for Adaptive Management Project Finance M&E Design at Entry and Implementation Performance of Implementing and Executing Entities PROJECT RESULTS Overall Results Component 1: Institutional Capacity Development focusing on Strengthening the Institute of Urban Transport Component 2: Individual Capacity Development through Training of Trainers and Professionals involved in Urban Transport Component 3: Preparation of Manual, Toolkits and Standards on various aspects of Urban Transport Component 4: Promotion, awareness raising and dissemination of information to expand and enhance the impacts of GEF SUTP Relevance Effectiveness and Efficiency Country Ownership and Drivenness Mainstreaming Sustainability of Project Outcomes Terminal Evaluation i June 2017

3 UNDP Government of India Terminal Evaluation of Sustainable Urban Transport Project (Comp.1A) Impacts CONCLUSIONS, RECOMMENDATIONS AND LESSONS CORRECTIVE ACTIONS FOR THE DESIGN, IMPLEMENTATION, MONITORING AND EVALUATION OF THE PROJECT ACTIONS TO FOLLOW UP OR REINFORCE INITIAL BENEFITS FROM THE PROJECT PROPOSALS FOR FUTURE DIRECTIONS UNDERLINING MAIN OBJECTIVES BEST AND WORST PRACTICES IN ADDRESSING ISSUES RELATING TO RELEVANCE, PERFORMANCE AND SUCCESS APPENDIX A MISSION TERMS OF REFERENCE FOR THE FINAL EVALUATION OF COMPONENT 1A OF THE SUTP PROJECT APPENDIX B MISSION ITINERARY (FOR JANUARY 2017) APPENDIX C LIST OF PERSONS INTERVIEWED APPENDIX D LIST OF DOCUMENTS REVIEWED APPENDIX E PROJECT PLANNING MATRIX (PPM) FOR COMPONENT 1A OF SUTP (FROM JULY 2013) APPENDIX F - TRACKING TOOL APPENDIX G - SUTP EVALUATION QUESTIONS APPENDIX H - EVALUATION CONSULTANT AGREEMENT FORM Terminal Evaluation ii June 2017

4 UNDP Government of India Terminal Evaluation of Sustainable Urban Transport Project (Comp.1A) SYNOPSIS Title of UNDP supported GEF financed project: India: Sustainable Urban Transport Project (SUTP) UNDP Project ID: PIMS 3214 GEF Project ID: 3241 Evaluation time frame: April 2010 to January 2017 CEO endorsement date: April 15, 2010 Project implementation start date: April 15, 2010 Project end date: March 31, 2018 Date of evaluation report: March 31, 2017 Region and Countries included in the project: India GEF Focal Area Objective: SP-5 (for GEF-4): Promoting sustainable innovative systems for urban transport Implementing partner and other strategic partners: Development (MoUD) Implementing partner: Ministry of Urban Evaluation team members: Mr Roland Wong, International Consultant Dr. Sudhakar Yedla, National Consultant Acknowledgements: The Evaluators wish to acknowledge with gratitude the time and effort expended by all project participants and stakeholders during the course of the Sustainable Urban Transport Project Terminal Evaluation. In particular, we wish to thank the UNDP India, the Ministry of Urban Development, the Institute of Urban Transport as well as other former Project managers and former Project personnel for making the efforts to recall details of their time while on the project. Thank you again to all those we met during the mission for your hospitality and insights, and most importantly, your passion for sustainable urban transport and your visions of improved urban mobility in India. We sincerely hope that this report contributes towards a lower carbon future for urban transport throughout India. Terminal Evaluation iii June 2017

5 UNDP Government of India Terminal Evaluation of Sustainable Urban Transport Project (Comp.1A) EXECUTIVE SUMMARY This report summarizes the findings of the Terminal Evaluation Mission conducted during the January 26- February 3, 2017 period for the Component 1A of the GEF project: Sustainable Urban Transport Project (hereby referred to as SUTP or the Project), where UNDP received a US$4.05 million grant from the Global Environmental Facility (GEF) in April 2010 as part of a larger GEF grant managed under the World Bank for Components 1B and 2. Project Summary Table Project Title: GEF Project ID: 3241 Sustainable Urban Transport Project (for Component 1A only) at endorsement (Million US$) at completion (Million US$) UNDP Project GEF financing: 3214 ID: Country: India IA/EA own: 0. Region: Asia and the Pacific Government: Focal Area: Climate Change Other: 0 0. FA Objectives, (OP/SP): Executing Agency: Other Partners involved: SP5 for GEF 4: Promoting sustainable innovative systems for urban transport Ministry of Urban Development (MoUD) Total cofinancing: Total Project Cost: ProDoc Signature (date project began): September 2011 (Operational) Closing Date: Proposed: 31 December 2014 Actual: 31 March 2018 Project Description The objective of the overall GEF-supported SUTP is the reduction of the growth trajectory of GHG emissions from the transport sector in India through the promotion of environmentally sustainable urban transport, strengthening government capacity to plan, finance, implement, operate and manage climate friendly and sustainable urban transport interventions, and increasing the modal share of environmental friendly transport modes in project cities. The overall SUTP has 2 components, one for capacity development initiatives that is jointly managed by UNDP and the World Bank, and another component comprising of support for 4 demonstration projects in certain selected cities and managed by the World Bank. This Terminal Evaluation (TE) covers the capacity development initiatives managed by UNDP under Component 1A that were designed to enable national capacity through the Government of India s Ministry of Urban Development (MoUD) to support wide-scale development of sustainable urban transport (SUT) through large urban centers of India in Component 2. During the period when the SUTP Project was being prepared in 2009, India was experiencing surging economic growth and increased urbanization leading to a rise in the ownership and use of motorized 1 As per the revised allocation of the GoI as approved by Standing Committee. This is not contained in the SUTP ProDoc. Terminal Evaluation iv June 2017

6 UNDP Government of India Terminal Evaluation of Sustainable Urban Transport Project (Comp.1A) vehicles in its cities. The increased number of motorized vehicles for travel within India s cities has been exacerbated by the availability of lower priced cars, increasing the share of motorized trips within urban areas where previously travel modes such as walking, cycling and the use of buses had been more dominant. This shift in travel modes has had an adverse impact on the urban quality of life through widespread congestion, increased air pollution from traffic, and increased GHG emissions from India s urban transport sector. The response by the Government of India to these issues of urban congestion has been the adoption of a National Urban Transport Policy (NUTP) in April The NUTP placed an emphasis on improving the livability of India s cities for sustainable urban growth focusing on improved urban mobility rather than on transport infrastructure. At the commencement stages of Component 1A of SUTP, full adoption and implementation of NUTP would not have been possible without participation of local stakeholders. To support local implementation (through World Bank implemented components of SUTP) for developing pilot sustainable urban transport projects, Component 1A of SUTP sought to address the following issues: highly fragmented institutional arrangements for urban transport in India in 2010; weak coordination between national, state and municipal sectors where systematic procedures on public consultation and technical guidelines for planning, preparing, appraising and monitoring and evaluating urban transport investments are largely ineffective; inadequate capacities in several state and municipal institutions to address urban transport issues in a comprehensive and collaborative environment; absence of local capacity to effectively collect knowledge on local transport issues that can be used within a framework of best practices by transport practitioners to plan, design, operate and manage urban transport investments; lack of formal feedback mechanisms between the public and providers of public transport; and the absence of high-quality urban transport projects that could demonstrate best practices and the benefits of sustainable urban transport, as well as cultivating changes in the way urban transport is planned, operated and maintained. Project Results Overall results of Component 1A of the SUTP Project can be summarized through actual outcomes of Component 1A activities. Table A: Comparison of Intended Project Outcomes from the Inception Report to Actual Outcomes Intended Outcomes in April 2010 ProDoc Overall Project Goal (Impact): Government capacity strengthened to plan, finance, implement, operate and manage climatefriendly and sustainable urban transport interventions at national, state and city levels Actual Outcomes as of March 2017 Actual Overall Project Impact: Government capacity has been strengthened through a strengthened Institute of Urban Transport (IUT) as a focal and quasi-independent organization that represents the Government on SUT issues including the appraisal of SUT investments. This in turn has led to the scale up and development of SUT projects in pilot cities under the SUTP Project in India, and an IUT that represents the GoI to provide handholding of city and state governments in the design of SUT systems in a number of cities to an advanced design or investment stage. Terminal Evaluation v June 2017

7 UNDP Government of India Terminal Evaluation of Sustainable Urban Transport Project (Comp.1A) Intended Outcomes in April 2010 ProDoc Project s Goal (Outcome): IUT is recognized by states and cities as a national urban transport knowledge centre Outcome 1: Institute of Urban Transport strengthened so as to provide substantial support to local governments in implementing the National Urban Transport Policy Outcome 2: Government officials, urban planners, practitioners receive training on various aspects of sustainable urban transport. Outcome 3: Manuals, Toolkits and Standard prepared to serve as reference documents, guides to develop and implement of sustainable urban transport. Outcome 4: Increased awareness of Sustainable Urban Transport interventions among city government officials and transport sector professionals. Actual Outcomes as of March 2017 Actual Project Outcome: IUT is developing into a national urban transport knowledge center through the provision of technical assistance to 6 states in implementing various provisions of the NUTP, and training and advisory services to 14 states/cities in implementing various provisions of the NUTP. Actual Outcome 1: IUT has been strengthened, and provides substantial support to local governments in implementing the NUTP. This has included following the recommendations of an IUT business plan that included a methodical process to expand its operations, functions and services to meet a demand for "handholding" technical assistance to local governments Actual Outcome 2: Government officials, urban planners, and urban transport professionals received training on various aspects of sustainable urban transport through 10 training programs and 10 annual workshops held throughout India. Actual Outcome 3: Manuals and toolkits were prepared to serve as reference documents, and guides to develop and implement SUTP projects throughout India, and serve as a basis for setting national SUT standards. The PMO, however, was not in a state of readiness to prepare such standards due to the lack of successful and operational SUTP projects in India during the implementation of Component 1A between 2010 and Actual Outcome 4: There is increased awareness of sustainable urban transport mentions amongst city and state government officials as well as urban transport sector professionals. A strong indicator of this outcome is the involvement of more than 60 cities in India on planning SUTP initiatives. Summary of Conclusions, Recommendations and Lessons As a consequence of implementing Component 1A of the SUTP, the capacity in India to promote sustainable urban transport as well as design and implement SUT projects has significantly improved in the 5 participating cities of SUTP. While this is a satisfactory achievement, the level of capacity built in India is not at a level where the number of transport professionals can fully satisfy the demand for planning and design of SUT projects in India, nor is there a critical mass of urban transport practitioners to grow the profession to meet this demand. IUT's relationship with other active urban transport institutions in India and its strategy for interaction with these institutions is not clear. Interaction with these other institutes (such as NIUA, CEPT, IIT Delhi, TERI, IGIDR, CIRT and IUA) may be beneficial to the acceleration of IUT's development as the GoI's premier agency on appraising SUTP proposals and developments. Current issues to the sustainability of IUT are mainly related to the pending approval of a financial corpus that will sustain IUT s operations after the EOP. Without financing support, IUT will experience understaffing to sustain current levels of research and KMC management; uncertainty in its long-term sustainability; and low morale resulting from this uncertainty. Another important achievement of Component 1A has been the establishment of a Knowledge Management Database Centre (KMC) to manage transport-related data from local governments and Terminal Evaluation vi June 2017

8 UNDP Government of India Terminal Evaluation of Sustainable Urban Transport Project (Comp.1A) NGOs that is required for effective design of SUT projects. However, actual collection of transport-related data focused on more than 40 cities, diluting pilot efforts to collect comprehensive transfer related data from the 5 pilot cities. In many of these cities, the quality of data was poor that required extensive analysis and efforts to integrate with the KMC s new SUT database, and there were missing data sets due to the lack of capacity or knowledge at the local government levels for this data such as energy-related data of urban transport. Finally, the insertion of UNDP on a World Bank-GEF project for capacity building has worked well, and served as a good foundation to enable IUT to improve the capacity of local government level personnel to design and implement SUT projects. This has subsequently improved the capacities of local governments to design and implement demonstration projects under Component 2. Corrective actions for the design, implementation, monitoring and evaluation of the project: Action 1 (to UNDP): For projects that involve two implementing agencies, assurances should be made that each implementing agency has its own project planning matrix for the monitoring and evaluation of its own activities. Action 2 (to UNDP and MoUD): A more detailed plan and vision of IUT should have been prepared as a target for implementers of Component 1A. Action 3 (to UNDP and MoUD): Improving the strengthening and increasing the growth of the KMC could have been realized if strategically planned with the collection of transport-related data (including energyrelated data) that focused on fewer cities, such as the 5 pilot SUT cities. This would have included a process to formulate a business plan for the KMC with a renewed focus on a fewer cities where an effective urban transport data collection system could be piloted. Actions to follow up or reinforce initial benefits from the project: Action 4 (to MoUD and UNDP): IUT needs to prepare a strong and updated business plan that proposes collaborative mechanisms with and complements the work of other existing institutions involved with urban transport in India. Action 5 (to MoUD and UNDP): Training materials for government personnel as well as to urban transport professionals needs to become more comprehensive and broad-based. This would include topics for training such as energy savings and GHG emission reductions from SUT projects, construction techniques, construction management, and the maintenance of SUT-related infrastructure. Action 6 (to MoUD and UNDP): Closely monitor the progress of the approval of the MoUD financial proposal to Cabinet for a financial corpus of USD 1.5 million (INR 10 crore) to be used for post-undp operation and administration of IUT and its management of the KMC. Proposals for future directions underlining main objectives of Component 1A of SUTP: Action 7 (to UNDP and the World Bank): Act on the request of MoUD for a continuation of donor supported capacity building of SUT professionals throughout India. This would include the updating all training modules; expanding topics in training modules such as construction contracting, construction management, construction techniques and use of special building materials in SUT projects; energy Terminal Evaluation vii June 2017

9 UNDP Government of India Terminal Evaluation of Sustainable Urban Transport Project (Comp.1A) savings and GHG reduction benefits of SUT projects including new topics such as bus operator training on eco-driving habits; more emphasis on SUT case studies from different regions of India as well as other countries; specialized training programs; and a continuation of training Master Trainers through IUT Best and worst practices in addressing issues relating to relevance, performance and success: Poor practice: Implementation of Component 1A contained activities related to national research such as municipal level data collection for sustainable urban transport purposes, an activity which expanded from 5 cities to more than 40 cities, diffusing efforts to effectively manage data for the KMC. Poor practice: With a mandate to build the capacity of IUT as the main entity to advise MoUD on all issues related to SUT development in India, the PMO could have accelerated this process through networking with relevant national institutes on urban transport in addition to international institutes. Best practice: To ensure constant improvement of training programs, aggressive follow-up on feedback surveys of training programs is necessary. Best practice: An energetic and efficient project management unit is required to manage a large capacity building project where there are numerous consultations and approvals required to select attendees of various training programs, especially within the Indian Government system. Evaluation Ratings 2 1. Monitoring and Evaluation Rating 2. IA & EA Execution Rating M&E design at entry 3 Quality of Implementation Agency - 5 UNDP M&E Plan Implementation 4 Quality of Execution - Executing 5 Entity (MoUD) Overall quality of M&E 4 Overall quality of Implementation / 5 Execution 3. Assessment of Outcomes Rating 4. Sustainability 3 Rating Relevance 4 2 Financial resources 4 Effectiveness 5 Socio-political 4 Efficiency 5 Institutional framework and 4 governance Overall Project Outcome Rating 5 Environmental 4 Overall likelihood of sustainability 4 2 Evaluation rating indices (except sustainability see Footnote 2, and relevance see Footnote 3): 6=Highly Satisfactory (HS): The project has no shortcomings in the achievement of its objectives; 5=Satisfactory (S): The project has minor shortcomings in the achievement of its objectives; 4=Moderately Satisfactory (MS): The project has moderate shortcomings in the achievement of its objectives; 3=Moderately Unsatisfactory (MU): The project has significant shortcomings in the achievement of its objectives; 2=Unsatisfactory (U) The project has major shortcomings in the achievement of its objectives; 1=Highly Unsatisfactory (HU): The project has severe shortcomings in the achievement of its objectives. 3 Sustainability Dimension Indices: 4 = Likely (L): negligible risks to sustainability; 3 = Moderately Likely (ML): moderate risks to sustainability; 2 = Moderately Unlikely (MU): significant risks to sustainability; and 1 = Unlikely (U): severe risks to sustainability. Overall rating is equivalent to the lowest sustainability ranking score of the 4 dimensions. 4 Relevance is evaluated as follows: 2 = Relevant (R); 1 = Not relevant (NR) Terminal Evaluation viii June 2017

10 UNDP Government of India Terminal Evaluation of Sustainable Urban Transport Project (Comp.1A) ABBREVIATIONS Acronym Meaning APR-PIR Annual Project Report - Project Implementation Review CEPT Center for Environmental Planning and Technology University CIRT Central Institute of Road Transport CMP Comprehensive Mobility Plan CO UNDP Country Office CO2 Carbon Dioxide COE Centre of Excellence CP Country Programme CPAP Country Programme Action Plan CPWD Central Public Works Department DDA Delhi Development Authority DMRC Delhi Metro Rail Corporation DTC Delhi Transit Corporation EE Energy Efficiency EOP End-of-Project FY Fiscal Year FYP Five-Year Plan GDP Gross Domestic Product GEF Global Environment Facility GHG Greenhouse gas GIZ German International Technical Assistance GoI Government of India IGIDR Indira Gandhi Institute of Development Research IIT Indian Institute of Technology INR Indian Rupee ITDP Institute for Transportation and Development Policy (USA) IUT Institute of Urban Transport JnNURM Jawaharlal Nehru National Urban Renewal Mission JTPA Japan Transport Planning Association KMC Knowledge Management Center kwh kilowatt hour LFA Logical Framework Matrix LTA Land Transport Authority (Singapore) M&E Monitoring and Evaluation MoU Memorandum of Understanding MoUD Ministry of Urban Development Mtoe Million tonnes of oil equivalent MTR Mid-Term Review NAPCC National Action Plan on Climate Change NGO Non-governmental organization NIUA National Institute of Urban Affairs NPD National Project Director NPM National Project Manager NTDPC National Transport Policy Development Committee NUTP National Urban Transport Policy PAD Project Appraisal Document (World Bank) PIMS UNDP/GEF Project Information Management System Terminal Evaluation ix June 2017

11 UNDP Government of India Terminal Evaluation of Sustainable Urban Transport Project (Comp.1A) Acronym PIR PMO PPG PPM PRF ProDoc PSC SLB SMART SUT SUTP tco2 TE TERI ToR ToT UITP UMTC UNDAF UNFCCC UNDP USD Meaning Project Implementation Review Project Management Office (within SUTP) Project Preparatory Grant (GEF) Project Planning Matrix Project Results Framework UNDP Project Document Project Steering Committee Service Level Benchmark Specific, Measurable, Attainable, Relevant and Time-bound Sustainable urban transport Sustainable Urban Transport Project Tonne of Carbon Dioxide Terminal Evaluation The Energy and Resources Institute Terms of Reference Training of trainers Union Internationale des Transports Public Urban Mass Transit Company Limited (Gurgaon, Haryana) UN Development Assistance Framework UN Framework Convention on Climate Change UN Development Programme United States dollar (= 64 Indian Rupee) Terminal Evaluation x June 2017

12 1. INTRODUCTION 1. This report summarizes the findings of the Terminal Evaluation Mission conducted during the January 26 to February 2, 2017 period for the GEF-financed Project entitled: Sustainable Urban Transport Project (SUTP) where UNDP received a USD 4.05 million grant from the Global Environmental Facility (GEF) to manage Component 1A. In this Terminal Evaluation (TE) report, the UNDP-managed portion of SUTP is referred to as Component 1A or the Component 1A of SUTP. 2. The objective of the overall GEF-supported SUTP is the reduction of the growth trajectory of GHG emissions from the transport sector in India through the promotion of environmentally sustainable urban transport, strengthening government capacity to plan, finance, implement, operate and manage climate friendly and sustainable urban transport interventions, and increasing the modal share of environmental friendly transport modes in project cities. The overall SUTP has 2 components, one for capacity development initiatives that is jointly managed by UNDP and the World Bank, and another component comprising of support for 4 demonstration projects in certain selected cities and managed by the World Bank. This TE covers the capacity development initiatives managed by UNDP under Component 1A that were designed to enable national capacity through the Government of India s Ministry of Urban Development (MoUD) to support wide-scale development of sustainable urban transport (SUT) through large urban centers of India in Component Purpose of the Evaluation 3. In accordance with UNDP and GEF M&E policies and procedures, all full and medium-sized UNDP support GEF financed projects are required to undergo a Terminal Evaluation (TE) upon completion of implementation of a project to provide a comprehensive and systematic account of the performance of the completed project by evaluating its design, process of implementation and achievements vis-à-vis GEF project objectives and any agreed changes during project implementation. As such, the TE for Component 1A of the SUTP Project serves to: promote accountability and transparency, and to assess and disclose levels of Project accomplishments; synthesize lessons that may help improve the selection, design and implementation of future GEF activities; provide feedback on recurrent issues across the portfolio, attention needed, and on improvements regarding previously identified issues; contribute to the GEF Evaluation Office databases for aggregation, analysis and reporting on effectiveness of GEF operations in achieving global environmental benefits and on the quality of monitoring and evaluation across the GEF system. 4. This TE was prepared to: be undertaken independent of management of Component 1A of SUTP to ensure independent quality assurance; apply UNDP-GEF norms and standards for evaluations; assess achievements of outputs and outcomes, likelihood of the sustainability of outcomes, and if Component 1A of SUTP met the minimum M&E requirements; Terminal Evaluation 1 June 2017

13 report basic data of the evaluation of Component 1A, as well as provide lessons from Component 1A on broader applicability. This would include an outlook and guidance in charting future directions on sustaining current efforts by UNDP, the Government of India, and their donor partners to building capacity to plan and implement sustainable urban transport. 1.2 Scope and Methodology 5. The scope of the TE for Component 1A of SUTP was to include all activities funded by GEF and activities from parallel-financing. The Terms of Reference (ToRs) for the TE are contained in Appendix A. Key issues addressed on this TE include: Design of Component 1A and its effectiveness in achieving the stated aims of promoting environmentally sustainable urban transport in India and to improve the usage of environmentally friendly transport modes in project cities; Assessment of key financial aspects of Component 1A, including the extent of co-financing planned and realized; The effectiveness of Component 1A in building the capacities of target urban transport institutions and of sustainable urban transport personnel; Strengths and weaknesses of Component 1A implementation, monitoring and adaptive management and sustainability of Component 1A outcomes including the exit strategy of Component 1A; Results and impacts of the implemented activities including views from Component 1A focal points (and other relevant stakeholders) on the impacts of capacity building activities implemented under Component 1A and their recommendations on the future initiatives; and Recommendations, lessons learned, best practices from implementing Component 1A of SUTP that could be used on other similar GEF projects. 6. Outputs from this TE will provide an outlook and guidance in charting future directions on sustaining current efforts by UNDP, and the Government of India, to sustaining capacity building efforts for personnel developing, maintaining and operating sustainable urban transport systems. 7. The methodology adopted for this evaluation includes: Review of Component 1A documentation (i.e. APR/PIRs, mid-term review (MTR), meeting minutes of Project Steering Committee or multipartite meetings) and pertinent background information; Interviews with key personnel of Component 1A including the current and former Project Managers, technical advisors (domestic and international), and SUTP Project developers; and Interviews with relevant stakeholders including other government agencies, beneficiaries and training participants from various cities throughout India. A detailed itinerary of the Mission is shown in Appendix B. A full list of people interviewed and documents reviewed are given in Appendix C and Appendix D respectively. The Evaluation Mission for the UNDP-GEF supported Component 1A was comprised of one international expert and one national expert. Terminal Evaluation 2 June 2017

14 8. Component 1A of SUTP was evaluated for overall results in the context of: Relevance the extent to which the outcome is suited to local and national development priorities and organizational policies, including changes over time; Effectiveness the extent to which an objective was achieved or how likely it is to be achieved; Efficiency the extent to which results were delivered with the least costly resources possible; and Sustainability - The likely ability of an intervention to continue to deliver benefits for an extended period of time after completion. 9. All possible efforts have been made to minimize the limitations of this independent evaluation. Notwithstanding that 5 days were spent in New Delhi during the period of January 26 th to February 2 nd, 2017 by the evaluation team to collect and triangulated as much information as possible, followup interviews and Skype conversations by the evaluation team were also made after the New Delhi mission. 1.3 Structure of the Evaluation 10. This evaluation report is presented as follows: An overview of Component 1A design; A review and critique of the Project Planning Matrix (PPM) for Component 1A; A review of the design of Component 1A; A review of Component 1A implementation from commencement of operations in April 2011 to the present including monitoring and evaluation systems; An assessment of Component 1A results based on Component 1A objectives and outcomes through relevance, effectiveness and efficiency criteria; Assessment of sustainability of Component 1A outcomes; and A presentation of conclusions, recommendations, and best and worst practices that can be used as lessons learned. 11. This evaluation report is designed to meet GEF s Guidelines for GEF Agencies in Conducting Terminal Evaluations, Evaluation Document No. 3 of 2008: The Evaluation also meets conditions set by: the 2012 UNDP Document entitled UNDP GEF Terminal Evaluation Guideline : the UNDP Document entitled Handbook on Planning, Monitoring and Evaluating for Development Results, 2009: and the Addendum June 2011 Evaluation : June-2011.pdf Terminal Evaluation 3 June 2017

15 2. PROJECT DESCRIPTION AND DEVELOPMENT CONTEXT 2.1 Project Start and Duration 13. Component 1A of the Sustainable Urban Transport Project officially commenced implementation on April 15, 2010, the date when the Indian government signature for the Component 1A project document (ProDoc) was obtained. The duration of Component 1A was originally planned for 4 years ending in October In June 2013, the Mid-Term Review recommended that an extension for the Project for another 2 years, 2 months ending in December This request for project extension was approved in December Problems that Component 1A of SUTP Sought to Address 14. The ProDoc for the UNDP-implemented portion of the Sustainable Urban Transport Project Document (or Component 1A ProDoc) provides details on the need to address the institutional challenges and address gaps in competent individual transport planning skills through a number of national level capacity building initiatives. During the period when the Project was being prepared in 2009, India was experiencing surging economic growth and increased urbanization leading to a rise in the ownership and use of motorized vehicles in its cities. The increased number of motorized vehicles for travel within India s cities has been exacerbated by the availability of lower priced cars, increasing the share of motorized trips within urban areas where previously travel modes such as walking, cycling and the use of buses had been more dominant. This shift in travel modes has had an adverse impact on the urban quality of life through widespread congestion, increased air pollution from traffic, and increased GHG emissions from India s urban transport sector. 15. The response by the Government of India to these issues of urban congestion has been the adoption of a National Urban Transport Policy (NUTP) in April The NUTP placed an emphasis on improving the livability of India s cities for sustainable urban growth focusing on improved urban mobility rather than on transport infrastructure. At the commencement stages of Component 1A of SUTP, full adoption and implementation of NUTP would not have been possible without participation of local stakeholders. 16. In the context of capacity building to support local implementation (through World Bank implemented components of SUTP) for developing pilot sustainable urban transport projects, Component 1A of SUTP sought to address the following issues: highly fragmented institutional arrangements for urban transport in India in 2010; weak coordination between national, state and municipal sectors where systematic procedures on public consultation and technical guidelines for planning, preparing, appraising and monitoring and evaluating urban transport investments are largely ineffective; inadequate capacities in several state and municipal institutions to address urban transport issues in a comprehensive and collaborative environment; absence of local capacity to effectively collect knowledge on local transport issues that can be used within a framework of best practices by transport practitioners to plan, design, operate and manage urban transport investments; lack of formal feedback mechanisms between the public and providers of public transport; and Terminal Evaluation 4 June 2017

16 the absence of high-quality urban transport projects that could demonstrate best practices and the benefits of sustainable urban transport, as well as cultivating changes in the way urban transport is planned, operated and maintained. 2.3 Immediate and Development Objective of Component 1A of SUTP 17. The objective of Component 1A of SUTP was to strengthen Government capacity to plan, finance, implement, operate and manage climate-friendly and sustainable urban transport interventions at national, state and city levels. This was to be achieved mainly through the strengthening of the institute of Urban Transport (IUT), an entity under the management of MoUD. The Project Planning Matrix (PPM) for Component 1A of SUTP was formulated in July 2013 as a recommendation from the MTR and is contained in Appendix E. 2.4 Baseline Indicators Established 18. Objective-level baseline indicators of Component 1A includes: Number of cities that have MoUD-approved Comprehensive Mobility Plans (CMPs), which have been appraised by IUT for making investment in sustainable urban transport services (EOP target: 65); Number of states that IUT provides technical assistance in implementing various provisions of national urban transport policy (EOP target: 7); Number of cities for which IUT provides training and advisory services in implementing various provisions of national urban transport policy (EOP target: 10). The baseline value for all these indicators at the start of the Component 1A was zero. It is noteworthy that there were no GHG emission reduction targets for Component 1A of SUTP. With Component 1A focused solely on capacity building, the strengthened institutions including IUT and municipal level stakeholders were to be better equipped to implement Component 2 or pilot SUT projects at the local level, thereby generating GHG emissions. As such, GHG emission reduction benefits from the overall SUTP are to be monitored and reported in the evaluation of the World Bank-implemented portion of SUTP. 19. Outcome-level baseline indicators for Component 1A of SUTP includes: For Outcome 1, IUT follows a methodical process to expand its operations, functions and services based on a sound rationale with an EOP target of Business Plan prepared and recommendation implemented by IUT with financial support of MoUD ; For Outcome 2, IUT conducts 10 training programs and 10 workshops annually across India that provides training to at least 500 urban transport planners and practitioners with an EOP target of at least 25 manuals, toolkits and standards are developed under the project ; For Outcome 3, Manuals, Toolkits and Standards developed and disseminated to central, state and urban government departments and agencies with an EOP target of at least 25 manuals, toolkits and standards are developed under the project ; For Outcome 4, increasing number of cities demand services to plan, implement and operate environment friendly and sustainable urban transport interventions with an EOP target of compilation of experience, knowledge and insights shared with government officials. Terminal Evaluation 5 June 2017

17 There are also output level baseline indicators as well as outcome and objective level indicators of Component 1A which can be found in the July 2013 PPM in Appendix E. 2.5 Main Stakeholders 20. The primary stakeholders of Component 1A are the Ministry of Urban Development (MoUD), the Institute of Urban Transport (IUT), and selected municipal governments, especially those who are planning and implementing pilot SUT investment projects. Details of Component 1A stakeholders are provided in Section (Paras 42-43). 2.6 Expected Results 21. To achieve the specific Component 1A goals of Government capacity strengthened to plan, finance, implement, operate and manage climate-friendly and sustainable urban transport interventions at national, state and city levels and IUT is recognized by states and cities as a national urban transport knowledge centre, Component 1A was designed with the following expected Project outcomes (from the 2013 LFA): Outcome 1: Institute of Urban Transport strengthened to provide substantial support to local governments in implementing the National Urban Transport Policy; Outcome 2: Government officials, urban planners, practitioners received training on various aspects of sustainable urban transport; Outcome 3: Manuals, toolkits and standards prepared to serve as a reference documents, guide to develop and implement of sustainable urban transport; Outcome 4: Increased awareness of sustainable urban transport interventions amongst city government officials and transport sector professionals. Terminal Evaluation 6 June 2017

18 3. FINDINGS 3.1 Project Design and Formulation 22. Design of the overall SUTP Project was conducted during the period of The overall objective of the SUTP Project was to reduce the growth trajectory of GHG emissions from the transport sector through the promotion of environmentally sustainable urban transport, strengthening government capacity to plan, finance, implement, operate and manage climate friendly and sustainable urban transport interventions, and increasing the moral share of environmentally friendly transport modes in selected cities. The SUTP Project was to be implemented within 2 components: a component on national capacity development initiatives jointly managed by UNDP (Component 1A) and the World Bank (Component 1B), and the World Bank-managed Component 2 on demonstration SUT projects in selected cities. 23. The UNDP SUTP ProDoc provides details on Component 1A activities to develop national capacity for planning, implementing and managing SUT projects in India. The ProDoc also identifies a number of problems (in Para 16 of the ProDoc) that serve as barriers that prevent the resolution of urban congestion issues that currently plagues Indian cities. The strategy of the Component 1A of SUTP to overcome these barriers includes: Initiating, building and consolidating strong functional long-term partnerships between the federal Government of India (GoI) and local state and municipal level governments on SUTP development; enhancing the capacity of relevant stakeholders in urban transport to plan, implement, operate and manage SUT systems. This would include policymakers, planners, researchers, service providers, managers amongst other professionals involved in urban transport in India; and the creation of a national resource centre for urban transport to facilitate knowledge and information exchanges. 24. A Project Results Framework (PRF) for the Component 1A of SUTP was subsumed within the overall SUTP project PRF that was contained in the World Bank s Project Appraisal Document (PAD) of For UNDP management of Component 1A, this PRF could not be used as a basis for managing the UNDP Component 1A for a number of reasons 5 with the outcome that UNDP managed this component during the period without a clear set of definable impacts and outcomes. As a follow-up to the MTR recommendation of June 2013, a separate Project Planning Matrix (PPM) for Component 1A of SUTP was formulated in July Analysis of Project Planning Matrix 25. As mentioned in Para 24, a PPM for Component 1A was prepared in July As shown in Appendix E, the PPM for Component 1A of SUTP provides 7 outcome level indicators and targets (3 goal/objective level and 4 outcome level) and 19 output level indicators and targets to guide implementation of Component 1A towards its objective of Government capacity strengthened to plan, finance, implement, operate and manage climate-friendly and sustainable urban transport interventions at national, state and city levels. While the intent of the PPM and its description of 5 No overview of UNDP-implemented activities, preliminary sets of outcomes that required more discussion, indicators that only focused on day-to-day activities, and a lack of integration between the UNDP and World Bank components. Terminal Evaluation 7 June 2017

19 indicators is reasonably clear in terms of its targets, wording of most of the indicators and targets do not meet SMART criteria 6 and best practices for preparing PPMs. While a large number of comments can be made on the language of the PPM, some specific comments includes: The description of the Project outcomes are more oriented to Component 1A outputs rather than intended Component 1A outcomes. For example, Outcome 2 could have been described as government officials, urban planners, practitioners are enabled to prepare investmentlevel sustainable urban transport plans. Similarly, Outcome 3 could have been described as sustainable urban transport plans being prepared on the basis of available and updated manuals, toolkits and reference guides that were based on best international practices. The description of indicators are actually sub-outcomes and not indicators that describe a quantified output. For example, on the overall project goal (impact), the indicator could have been number of cities with MoUD approved Comprehensive Mobility Plan (CMP). This would be clearer in terms of what is being monitored for progress; The evaluators believe there is no necessity to provide outcome level indicators and targets, since the outcome itself should describe the target. Moreover, the outputs under a particular outcome should contribute towards the achievement of the outcome; All output descriptions should be described as outputs, not outcomes. For example, Output 1.2 can be described as IUT accreditation for Sustainable Urban Transport, or Output 1.3 described as a functional knowledge management data centre at IUT. The targets for each of these indicators would remain as 1 ; 26. One important comment on the PPM involves the desired outcomes for IUT at the conclusion of Component 1A. These are not provided and as such, there are issues emerging at the conclusion of UNDP s involvement on Component 1A on the sustainability and viability of IUT that have not been addressed. For example, there could have been outputs and indicators on IUT revenue or staffing levels which would also provide good indicators of the health and strength of IUT at the EOP Risks and Assumptions 27. The July 2013 PPM for Component 1A provides assumptions under which the proposed outcomes and outputs could be achieved. These assumptions also infer risks to the achievement of outcomes and outputs. Many of these assumptions are related to SUTP s implementing partner, MoUD, providing strong support and direction towards the use of the Institute of Urban Transport (IUT) as a national resource for developing SUT projects in large urban centres throughout India. Examples of the assumptions in the July 2013 PPM includes: MoUD provides one time financial corpus to IUT to maintain the required human and financial resources to function as Technical expert on Urban Transport for national, state and city governments ; MoUD appoints IUT as main agency to conduct policy research on urban transport ; IUT has adequate trained manpower to appraise Comprehensive Mobility Plans which is dependent on MoUD providing the one time financial corpus; MoUD directs state government to seek technical assistance and advisory service of IUT on various aspects of National Urban Transport Policy ; and 6 Specific, Measurable, Attainable, Relevant and Time-bound Terminal Evaluation 8 June 2017

20 Competent task-specific expertise is locally available for preparation of manuals, toolkits and MoUD accords approval to IUT to develop new standards. 28. The ProDoc, however, does contain an Off-line Risk Log for Component 1A in Annex 2 for which Component 1A management staff can manage risks against the goals of Component 1A and its intended outcomes. Examples of some of these risks includes: Institutional and capacity development under the project will not be sustained; Delays in project implementation and poor quality due to involvement of multiple cities and states whose capacity varies and whose commitments to the projects may change; Late or delayed provision of the one-time financial corpus to IUT from MoUD; and Demonstration projects will not be replicated in other cities and states without GEF grant support. These risks should also be listed with the PPM against the various Component 1A outcomes as a link to the Off-line Risk Log Lessons from Other Relevant Projects Incorporated into Component 1A Design 29. The ProDoc of Component 1A for SUTP does not list any other relevant projects into its design Planned Stakeholder Participation 30. The ProDoc does not contain any stakeholder participation plan, nor has the evaluation team been provided with any evidence of any strategic plan for stakeholder participation for Component 1A. Since the primary purpose of Component 1A was to increase the knowledge and build the capacity of selected stakeholders at the federal, state and municipal levels to plan, design, implement and manage sustainable urban transport initiatives, the ProDoc should have identified the stakeholders, especially considering the numerous federal agencies, state governments in India, and municipalities who would have a stake in improving urban mobility within India s busiest cities. This would have involved identification of priority federal agencies, and targeted selected cities in India for pilot SUTP initiatives. Moreover, there appears to be an absence of any analysis of stakeholders, which would assist in the prioritization of stakeholders for engagement Replication Approach 31. Component 1A did not have a specific replication approach to building capacity of all relevant stakeholders in India on sustainable urban transport. However, the need to transfer knowledge on sustainable urban transport in India to a wide spectrum of stakeholders implies the replication (or vertical scale-up) of knowledge transfer and training sessions to a large number of stakeholders. The UNDP ProDoc for Component 1A describes a focus on building the capacity of IUT to become a primary SUT resource centre in India that would serve the needs of urban transport professionals. This would include building the capacity of IUT to initiate capacity development programs. These IUT programs would provide training of trainer (ToT) programs for over 1,000 planners, policymakers, and urban transport professionals throughout all levels of government. The design of Component 7 This would include a stakeholder s interest in an SUT initiative, their sphere of influence on SUT, an assessment of impact if involved, and measures to mitigate obstacles to their involvement. Terminal Evaluation 9 June 2017

21 1A could have spread the focus onto additional institutions to assist IUT, many of whom could play a key role in replication (or a cascading effect) of the capacity building efforts. Excessive focus on IUT alone places higher risks of not achieving replication targets. 32. Furthermore, another replication aspect of Component 1A was the need to decentralize IUT functions through the development of regional IUT centres. Given the importance of local participation on any SUT initiative, activities of Component 1A were to include support from IUT to build the capacity of local institutions in selected urban centres throughout India (under Subcomponent 1.4). This aspect does not appear on the July 2013 PPM as a target UNDP Comparative Advantage 33. UNDPs comparative advantage to other donor agencies is its focus on policy-based and cross-sectoral approaches as well as building local capacities through effective collaboration with a wide range of local stakeholders (in this case, federal, state and municipal levels of government and the general public). This would include public and private sectors as well as technical experts, civil society and grassroots level organizations. These approaches are strongly applicable on national capacity building initiatives for SUT projects under Component 1A of SUTP. Given UNDP s long track record on a wide variety of projects within the transport and energy sectors, UNDP has earned a reputation as a credible and trusted partner to the GoI in capacity building initiatives and is suited as an implementing agency for Component 1A. Particularly in issues that require high quality technical inputs, UNDP has a strategic advantage as a partner and the method of deploying technical experts with GoI s line ministries has increased the effectiveness of GoI-UNDP initiatives Linkages between Component 1A of SUTP and Other Interventions within the Sector 34. Component 1A is a part of a larger SUTP project, the other components of which are being implemented through World Bank management. However, the UNDP ProDoc for Component 1A does not identify any linkages of capacity building initiatives with other transport-related interventions in India including the World Bank-managed Component 1B. This is unfortunate since the capacity building taking place on Component 1B is the dissemination of technical transport planning skills to cities and states and financing urban transport projects that contribute towards reducing emissions. As such, this linkage to Component 1B would provide an end point of an exit strategy to those implementing Component 1A Management Arrangements 35. The implementing partner of Component 1A is the Ministry of Urban Development in accordance with UNDPs National Implementation Modality (now referred to as National Execution or NEX modality. The NEX modality tasks MoUD with responsibility for certifying work plans and approved budgets, reporting on procurement, coordinating and tracking co-financing, terms of reference for contractors and tender documentation, and chairing the Project Steering Committee (PSC). The Chair of the PSC was to be the National Project Director (NPD) from MoUD. 36. The ProDoc also acknowledges that UNDP is a partner GEF agency for SUTP with the World Bank being the leading GEF agency, responsible for overall technical quality assurance for the entire SUTP Project. As such, UNDPs responsibilities to GoI were to ensure quality assurances in the implementation of national capacity building initiatives of Component 1A. These initiatives would Terminal Evaluation 10 June 2017

22 then feed into the implementation of selected pilot states and cities for sustainable urban transport managed by the World Bank, as well as tailored capacity development at the local level to enable local staff to implement demonstration SUT projects. PMO staff also managed implementation of the World Bank-implemented components of SUTP that provided seamless interaction between UNDP and World Bank supported activities. 37. An organogram of Component 1A implementation arrangements is provided on Figure 1. Figure 1: Current Management Arrangements for the UNDP-implemented Component 1A of the Sustainable Urban Transport Project (SUTP) Project Organisation Structure Project Steering Committee (PSC) Senior Beneficiaries State Governments Executive Secretary of MoUD Senior Supplier World Bank and UNDP Project Assurance World Bank UNDP (Component 1A) National Project Director (NPD) Project Management Office (PMO) within SUTP 1. National Project Manager 2. Training Specialist -National Agencies and Institutions -International Agencies and Consultants 3.2 Project Implementation 38. The following is a compilation of key events and issues of Component 1A of SUTP implementation in chronological order: Signing of the SUTP ProDoc by the Ministry of Urban Development and UNDP India was on April 15, 2010, marking the official start of Component 1A implementation; The IUT business plan was completed with Component 1A support in late 2011; Early activities of UNDP on Component 1A involved IUT organizing and drafting manuals and toolkits for training as well as dissemination activities at international and national conferences and the SUTP website; Terminal Evaluation 11 June 2017