ATM CBA Quality Checklist

Transcription

1 ATM CBA Quality Checklist Document information Project title Business Case Maintenance, Support and Coordination Project N Project Manager EUROCONTROL Deliverable Name ATM CBA Quality Checklist Deliverable ID D06-08 Edition Abstract The purpose of this Air Traffic Management (ATM) Cost Benefit Analysis (CBA) Quality Checklist is to provide a means to further improve the quality and consistency of ATM CBAs. It includes a list of points which need to be addressed at different stages of the CBA process such as before starting, during production and after completion. It also includes an initial differentiation for E-OCVM lifecycle phases V1, V2 and V3.

2 Authoring & Approval Prepared By Name & company Position / Title Date Kirsteen Purves / EUROCONTROL Project Team Member 10/12/10 Reviewed By Name & company Position / Title Date Nicolás Suárez Tetzlaff / AENA Project Member 22/11/10 Philippe Malinge / AIRBUS Project Member 22/11/10 Franco Scolaris / ALENIA Project Member 22/11/10 Flavien Gueudre / AVTECH Project Member 22/11/10 Lars Lindberg / AVTECH Project Member 22/11/10 Anne Baleras / EUROCONTROL Project Member 22/11/10 Hilde Craninx / EUROCONTROL Project Member 22/11/10 Vesna Cohen / EUROCONTROL Project Member 22/11/10 Marco Gibellini / EUROCONTROL Project Member 22/11/10 Paula Leal de Matos / EUROCONTROL Project Manager 22/11/10 Kirsteen Purves / EUROCONTROL Project Member 22/11/10 Rodolphe Salomon / EUROCONTROL Project Member 22/11/10 Agnieszka Wegner / EUROCONTROL Project Member 22/11/10 Vladimir Coca / INDRA Project Member 22/11/10 Mike Bennett / NATS Project Member 22/11/10 Tahir Latif / NATS Project Member 22/11/10 Nigel Tay / NATS Project Member 22/11/10 Herve Goutelard / Thales Project Member 22/11/10 Philippe Jasselin / Thales Project Member 22/11/10 Approved By Name & company Position / Title Date Paula Leal de Matos / EUROCONTROL Project Leader 17/12/10 Peter Martin / EUROCONTROL Sub WP 16.6 Leader 17/12/10 Mike Bennett / NATS Project Member 17/12/10 Philippe Malinge / AIRBUS Project Member 17/12/10 Philippe Jasselin / Thales Project Member 17/12/10 Vladimir Coca / INDRA Project Member 17/12/10 Nicolás Suárez Tetzlaff/ AENA Project Member 17/12/10 Franco Scolaris / ALENIA Project Member 17/12/10 1 of 15

3 Document History Edition Date Status Author Justification /12/10 Draft Kirsteen Purves Conversion of draft 0.4 from the previous deliverable template and inclusion of NATS comments /12/10 Revised Draft Kirsteen Purves Review by Partners /12/10 Final Kirsteen Purves Submission to SJU Copyright 2010 The European Organisation for the Safety of Air Navigation (EUROCONTROL) This document is published by EUROCONTROL for information purposes. It may be copied in whole or in part, provided that EUROCONTROL is mentioned as the source and to the extent justified by the non-commercial use (not for sale). The information in this document may not be modified without prior written permission from EUROCONTROL. 2 of 15

4 Table of Contents EXECUTIVE SUMMARY INTRODUCTION PURPOSE AND SCOPE OF THE DOCUMENT INTENDED AUDIENCE STRUCTURE OF THE DOCUMENT ACRONYMS AND TERMINOLOGY BACKGROUND INFORMATION KEY COMPONENTS OF A CBA LIFECYCLE PHASES ATM CBA CHECKLIST BEFORE STARTING A CBA DEFINING A CBA PERFORMING A CBA FINALISING A CBA CBA FEEDBACK LOOP REFERENCES...13 APPENDIX A E-OCVM LIFECYCLE PHASES...14 A.1 V1 - SCOPE...14 A.2 V2 - FEASIBILITY...14 A.3 V3 - PRE-INDUSTRIAL DEVELOPMENT & INTEGRATION of 15

5 List of tables Table 1: Acronyms and Terminology...6 Table 2: Example of table describing differences between V1, V2 and V3...8 Table 3: Stakeholder input differences between V1, V2 and V3...9 Table 4: Benefit and Cost Mechanism differences between V1, V2 and V Table 5: Input data differences between V1, V2 and V Table 6: Differences in results between V1, V2 and V List of figures Figure 1: Evolution of a CBA of 15

6 Executive summary The purpose of this Air Traffic Management (ATM) Cost Benefit Analysis (CBA) Quality Checklist is to provide a means to further improve the quality and consistency of ATM CBAs. It includes a list of points which need to be addressed at different stages of the CBA process such as before starting, during production and after completion. It also includes an initial differentiation for E-OCVM lifecycle phases V1, V2 and V3. 5 of 15

7 1 Introduction 1.1 Purpose and scope of the document The purpose of this Air Traffic Management (ATM) Cost Benefit Analysis (CBA) Quality Checklist is to provide a means to further improve the quality and consistency of ATM CBAs. The checklist provides a set of questions which should all be addressed by CBA practitioners for each ATM CBA they plan and conduct. The questions focus on the V1 to V3 lifecycle phases (which are described in Appendix A) and are grouped into the following topics: 1) Before Starting a CBA 2) Defining a CBA 3) Performing a CBA 4) Finalising a CBA 5) CBA Feedback Loop It is proposed that the checklist is used through all the stages of a CBA, including any iterations to refine the model and the input data. The checklist is intended to be used for CBAs at different levels, e.g. project or programme. 1.2 Intended audience This checklist is aimed at CBA practitioners who are already familiar with performing CBAs and who will conduct CBAs within SESAR and other European air traffic management projects/programmes. 1.3 Structure of the document This document has the following structure: Section 2 provides some background information on the key components of CBAs and the impact of the E-OCVM lifecycle phases. Section 3 contains the checklist. Section 4 provides the references. 1.4 Acronyms and Terminology Term Definition ANSP ATM CBA Air Navigation Service Provider Air Traffic Management Cost Benefit Analysis E-OCVM European Operational Concept Validation Methodology (Ref [2]) NATS NPV SESAR SJU United Kingdom Air Navigation Service Provider Net Present Value Single European Sky ATM Research Programme SESAR Joint Undertaking (Agency of the European Commission) Table 1: Acronyms and Terminology 6 of 15

8 2 Background Information 2.1 Key Components of a CBA The section provides an overview of what is meant by the term CBA. A CBA is an objective study in which the costs and benefits of a particular project are fully quantified in economic terms, taking full account of the times at which costs are paid and at which benefits accrue, for more details see Ref [1]. A CBA should: 1) apply discounted cash flow analysis 2) capture uncertain inputs using ranges and probabilities associated to the ranges 3) include sensitivity analysis: How do uncertainties influence the net present value (NPV) of the project? What are the most critical uncertainties to the NPV of the project? 4) include probabilistic risk analysis: What is the likelihood of having a certain NPV? The ease of including these key components and the reliability of the results should be considered when deciding which tool to use to perform CBAs. However this checklist can be applied independently of the tool being used. 2.2 Lifecycle Phases The E-OCVM Ref [2] defines eight lifecycle phases; V0 where ATM Needs are identified, V1 to V3 where solutions are identified and researched and V4 to V7 which cover the period from industrialisation to decommissioning of the solution. For more information on the lifecycle phases refer to Appendix A. This checklist has been defined for CBAs that are focusing on projects in lifecycle phases V1 to V3 which cover the research and development activities. It is expected that the CBA checklist will be used in each lifecycle phase and that the level of detail in the CBA will increase with each phase. End of V1 End ofv2 Endof V3 V1 CBA V2 CBA V3 CBA Figure 1: Evolution of a CBA 7 of 15

9 3 ATM CBA Checklist Ideally the checklist will be used throughout all the stages of a CBA, including any iterations to refine the model and the input data. However if your CBA is already underway it is advised that you still start at the first section and work through the questions in all the sections (do not just go straight to the section reflecting the current stage of your CBA). This is to ensure that all the points are addressed. Where differences between the requirements for V1, V2 and V3 CBAs have been identified they are provided in tables such as the one below. V1 V2 V3 Description for V1 CBA Description for V2 CBA Description for V3 CBA Table 2: Example of table describing differences between V1, V2 and V3 3.1 Before Starting a CBA Justification Is a CBA definitely needed in this case? Can it be justified? Would Cost-effectiveness analysis 1 or a more qualitative analysis be more appropriate? Consider the effort required to perform the CBA against the value of the project. Is thinking sufficiently developed to enable an effective CBA to be performed? Traceability Is this CBA a follow up or update to another CBA? If so, are the previous reports and data available to use as a starting point. 3.2 Defining a CBA Decision Is it clear what decision the CBA will support? Lifecycle Phase Which lifecycle phase (V1, V2 or V3) will the CBA support? Ideally the CBA will start in V1 and evolve along with the project. Stakeholders 2 Have the main impacted stakeholders been identified? 1 a method recommended when partners need to make a decision among various ways to achieve similar results. In this case costs of implementing a solution are usually quantifiable but benefits are difficult to quantify in monetary terms. This method will rank various options in terms of their cost per unit effectiveness. Ref [3] 2 Stakeholders are organizations and entities who will have to pay for or will be impacted by the project directly or indirectly. Examples in the ATM world are ANSP s, airspace users, airports, military, passengers, residents around an airport, etc. 8 of 15

10 Have they committed to provide the necessary inputs for all the lifecycle phases? V1 V2 V3 Is there a list of impacted stakeholders who should be involved? Are stakeholders involved in developing the CBA and providing input data? Is there a stakeholder plan describing how other the other stakeholders will be involved in V3? Have inputs been received from a wider set of stakeholders (than were involved in V2)? Has stakeholder buy-in been achieved? Table 3: Stakeholder input differences between V1, V2 and V3 Has the CBA process been used as a method to identify and understand stakeholder disagreements and any opposition, early in the project development? CBA focus Are all the important issues addressed? Does the CBA focus on what really matters? Alternatives Does the CBA identify and take account of alternatives to the project? Baseline and Scenarios Are the baseline and different scenarios identified and clearly described? Strategic Fit Does the CBA identify how this project fits in the strategy? Dependencies Are initial dependencies with other projects (overlaps, adverse effects, synergies) correctly identified? Assumptions Are the assumptions of the CBA transparent? Are the assumptions realistic? Are they based on the Standard Inputs (if applicable) Ref [4] Show-stoppers Are there any potential show-stoppers? Such as, disagreements between stakeholders or issues over enabler availability. 3.3 Performing a CBA Stakeholders Are the main impacted stakeholders pro-actively involved in performing the CBA? Are costs and benefits presented from the view point of each stakeholder? 9 of 15

11 Benefit and Cost Mechanisms Are the benefit and cost mechanisms 3 adequately explained for this life-cycle phase? Are potential negative effects included? V1 V2 V3 Are textual descriptions of the costs and benefits available? Are graphical representations (influence diagrams) of the costs and benefits available? Have these been reviewed with the stakeholders? Have the cost and benefit mechanisms been reviewed and updated with stakeholder involvement? Input Data Table 4: Benefit and Cost Mechanism differences between V1, V2 and V3 Have the data sources been recorded? Include names of experts, their expertise and their confidence in their judgements, details of documentation, etc. V1 V2 V3 Have potential data sources been identified? Have data sources been reviewed to see that they are the best available? Have alternative data sources been considered? Has the data been reviewed to see if available values have been updated or new sources identified? Table 5: Input data differences between V1, V2 and V3 Have the types of input data been recorded making a clear distinction between: Assumptions Expert Judgement Factual evidence Validation of Inputs and Outputs Are the inputs to the CBA consistent with the standard inputs and with other CBAs which have used similar inputs (e.g. cost of a certain type of equipment)? Are the inputs to the CBA adequately validated and evaluated? Considering the life-cycle phase of the project and the principle of getting things in proportion (e.g. there is no point in doing a CBA for a project which will cost as much as the project)? Are the outputs of the CBA supported by the outputs of other CBAs or other relevant studies? 3 Benefit mechanisms are a cause-effect description of the improvement proposed by the project. They show how benefits are delivered Ref [5]. Cost mechanisms are description of the potential costs of the project broken down into relevant cost categories (e.g. capital, one-off, operating) 10 of 15

12 Uncertainty Does the CBA identify and take into account the main uncertainties of the project, e.g. by using ranges for uncertain input data? Have sensitivity and risk analyses been carried out as part of the CBA? Dependencies / Double Counting 4 Are dependencies with other projects (overlaps, adverse effects, synergies) correctly identified? Have potential areas of double counting been discussed with other projects providing the same benefits/ incurring the same costs? Assumptions Results Have the assumptions been revised and updated? Have the results been realistically interpreted? V1 V2 V3 Do the results provide orders of magnitude, min/max values and broad ranges? How do the different scenarios compare? Does the CBA indicate that the project should continue into V2? What is the NPV of the different scenarios? How sensitive are the results? How do the different risk profiles compare? Does the CBA indicate that the project should continue into V3? What is the NPV of the chosen solution? How sensitive are the results? What is the associated risk profile? Does the CBA indicate that the project should continue into V4: Industrialisation? Iteration Table 6: Differences in results between V1, V2 and V3 Do the results indicate that the model and input data should be reviewed before the lifecycle phase CBA is delivered? 3.4 Finalising a CBA Stakeholders Have the main stakeholders impacted by the project been pro-actively involved in the CBA from the start in a continuous and consistent way? 4 Double Counting occurs when multiple projects all claim to provide the same type of benefit, e.g. will two projects each predicting a 5% increase in capacity lead to a 10% capacity increase? Also if the capacity need is only 5% and the first project will meet that need then the second project cannot lay claim to capacity benefits. 11 of 15

13 Conclusions and Recommendations Clarity Does the CBA report contain clear Conclusions and Recommendations taking stock of the results? Are there any issues, monetary or otherwise, raised during the CBA that need further investigation or review? Are the deliverables of the CBA clear to the audience? Are different deliverables aimed at different audiences? Is the CBA transparent? Is it clear how the results were achieved? Does the CBA describe and explain the costs/benefits that are not monetized, instead of just omitting them? Are the trade-offs in performance (e.g. additional cap and safety) mentioned in the CBA? (and in more detail in the Business Case, Ref [5]) Challengeable Is it possible to challenge the assumptions, inputs, outputs of the CBA? Deliverables Has the CBA report been reviewed by the impacted stakeholders? Have fully documented versions of the CBA models been provided in a well-known software package, i.e. EXCEL, along with the input data and the results? Does the deliverable mention that this checklist was used to guide the CBA? Model Review Has the model been reviewed and audited by someone who wasn t involved in its production? And is that documented? Replication Do the CBA deliverables provide all the necessary information and data to be possible to replicate it? 3.5 CBA Feedback Loop Feedback loop Is there a process in place to review the costs and benefits estimated in the CBA against the costs and benefits actually realised? 12 of 15

14 4 References [1] SESAR , ATM CBA for Beginners, D06-05, v , 2011 [2] European Commission/EUROCONTROL, European Operational Concept Validation Methodology (E-OCVM), Version 3.0, February 2010 [3] European Commission, Impact Assessment Guidelines, SEC(2009) 92, 15 January 2009 [4] EUROCONTROL, Standard Inputs for EUROCONTROL Cost Benefit Analyses, 07/04/03-18, V4.0, October 2009 [5] SESAR Early Project, Proposed Approach for SESAR Business Cases, Issue 1, 30 July of 15

15 Appendix A E-OCVM Lifecycle Phases A.1 V1 - Scope This lifecycle phase identifies the operational/technical solutions for meeting the target performance identified in V0, ATM Needs. The output from this lifecycle phase will be a V1 CBA. A.2 V2 - Feasibility The main objective of this phase is to develop and explore the individual concept elements and supporting enablers until the retained concept(s) can be considered operationally feasible or it can be established that further development is no longer justified. The output from this lifecycle phase will be a V2 CBA. A.3 V3 - Pre-industrial development & integration The objective of this phase is threefold: To further develop and refine operational concepts and supporting enablers to prepare their transition from research to an operational environment; To validate that all concurrently developed concepts and supporting enablers (procedures, technology and human performance aspects) can work coherently together and are capable of delivering the required benefits; To establish that the concurrent packages can be integrated into the target ATM system. The output from this lifecycle phase will be a V3 CBA. 14 of 15

16 -END OF DOCUMENT - 15 of 15