MAT Report Principles and Layout

Transcription

1 MAT Report Principles and Layout Document information Edition date 28th of January 2015 Edition number Abstract In order to provide a global view of the progress of Maturity within the SESAR Programme, a Maturity Assessment Report (MAT report) is proposed to be periodically issued, which provides a consolidated view of the progress towards maturity of all Operational Focus Areas (OFAs) and associated Operational Improvement Steps (OI steps). To support this approach, a Maturity Assessment Tool (MAT) has been developed which provides this consolidated view of OFAs and OI steps maturity, based on the individual expert judgment assessments of OI steps. Based on the SESAR Validation Maturity Criteria, MAT allows to asses OI step maturity at EOCVM transition Phases, V1->V2, V2->V3 and V3->V4 (European Operational Concept Validation Methodology). This document presents: the principles applied to determine the OI Step Maturity and the resulting progress towards OFA Maturity; the layout of the MAT report; user guidelines to help reading the MAT reports.

2 Authoring & Approval Prepared By - Authors of the document. Name & Company Position & Title Date Mark Watson / Industrial Support ATM Performance Expert 28/01/2015 Reviewed By - Reviewers internal to the project. Name & Company Position & Title Date Alfredo Gomez / SJU Head of Validation and Verification 28/01/2015 Approved for submission to the SJU By - Representatives of the company involved in the project. Name & Company Position & Title Date Gianni Inzerillo / Industrial Support SESAR IS Director 28/01/2015 Document History Edition Date Status Author Justification /04/2014 First Edition Mark Watson Definition of MAT report /09/2014 Second Edition Mark Watson General publication /01/2015 Third Edition Mark Watson General publication Intellectual Property Rights (foreground) This deliverable consists of SJU foreground. 2 of 13

3 Table of Contents 1 INTRODUCTION, SESAR VALIDATION MATURITY SESAR VALIDATION MATURITY CRITERIA SESAR VALIDATION MATURITY CRITERIA SATISFACTION OI STEP MATURITY ASSESSMENT PER VALIDATION PHASE PROGRESS TOWARDS OFA MATURITY MAT REPORT CONTENT REFERENCES OFA MATURITY OFA Code and Title OFA Progress Performance Assessment OI Step Maturity Summary OI STEP MATURITY OI Step Code and Title Overview of V1, V2 or V3 maturity Evidence of Validation OI Step Spider Chart List of figures Figure 1: OI Step Maturity Satisfaction Rating... 5 Figure 2: OI Step Satisfaction Scoring... 6 Figure 3: Average Satisfaction Calculation... 6 Figure 4: OI Step Assessment... 6 Figure 5: Overall OI Step Assessment... 6 Figure 6: Points allocation... 7 Figure 7: OI Step Maturity Calculation... 7 Figure 8: OFA Maturity Calculation... 7 Figure 9: OFA Maturity Example Calculation... 7 Figure 10: Reference Material (Example)... 8 Figure 11: OFA Summary (Example)... 8 Figure 12: OI Step Summary (Example) of 13

4 1 Introduction, The outcome of validation activities is assessed in the Release management process. Yearly, the exercises planned to complete the full maturity of an OI step (V3) are extracted out of the V&V Roadmap and consolidated into the Release Plan. On the other hand, the OFA roll-out process has introduced the need to assess the progress towards maturity of an OFA, in other words, the progress of maturity of all OI steps which belong to that OFA, all along their lifecycle (V1, V2, V3), allowing to achieve SESAR solutions. To answer these two needs, a Maturity Assessment Tool (MAT) has been developed, which provides a consolidated view of the maturity for each SESAR solution, based on individual OI step assessments performed by experts. Based on SESAR Validation Maturity Criteria, MAT allows assessing OI step maturity at V1->V2, V2- >V3 and V3->V4 1 transition phases (European Operational Concept Validation Methodology). A report (MAT report) is generated per Concept Storyboard Step using MAT, providing a consolidated view of the individual OI step assessments. Section 2 defines the principles applied to determine the OI step maturity and progress towards OFA maturity. Section 3 presents the layout of the MAT report and provides user guidelines to help reading the MAT reports. 1 In this Report, the transition criteria V1->V2 V2->V3, V3->V4 are referred as V1, V2 and V3 4 of 13

5 2 SESAR Validation Maturity 2.1 SESAR Validation Maturity Criteria The SESAR Validation Maturity Criteria guidance has been published (03/10/2012) and can be found in the Extranet Programme Library under Validation & Verification. The SESAR Validation Maturity Criteria are categorized in the following seven "assessment threads : 1. Operations e.g. Operational Requirements; 2. System e.g. System Requirements; 3. Performance e.g. Performance Assessments achieved ( i.e. Operational Performance such as Capacity, Safety, Security, Environmental, Human Performance); 4. Standards & Regulations (e.g. information to update standards is mature); 5. Transition (e.g. analysis to transition to V4 and industrialisation); 6. Programme (e.g. related OI steps and Enablers validated together); 7. Validation (e.g. quality of the validation and techniques used). The Maturity Assessment Tool (MAT) implements the SESAR Validation Maturity Criteria. 2.2 SESAR Validation Maturity Criteria Satisfaction Systems Engineering Reviewers or OFA Coordinators use MAT to assess the "satisfaction" of the SESAR Validation Maturity Criteria using the following satisfaction rating (Figure 1). Satisfaction Not Applicable Not Achieved Partially Complete Blocking Issues Partially Complete Non Blocking Fully Achieved Interpretation The criterion cannot be applied to the SESAR Solution, or, the criterion is not relevant to the SESAR Solution. The criterion has not been satisfied for the SESAR Solution, but should be applicable, or, the criterion is relevant to the SESAR Solution, but it has not been assessed. The remaining issues prevent progress of the SESAR Solution to the next maturity level until these issues have been assessed [Includes OI where further work is planned]. This may include future exercises planned to complete the validation. Remaining issues do not prevent progress to the next maturity level [at this time] (provided they are addressed as part of the activities in the next maturity phase) No issues are identified to resolve after the exit of the V-phase Figure 1: OI Step Maturity Satisfaction Rating A scoring (Figure 2) is allocated to the assessed satisfaction rating within a validation phase (V1, V2 or V3) in order to provide a measure of the average satisfaction rating of an assessment thread, and the average satisfaction rating of an OI step. Criteria assessed as Not Applicable are allocated No score, and therefore do not contribute to calculations 2. 2 If a satisfaction response has not been provided then it is assumed that the criteria has not been satisfied. 5 of 13

6 Satisfaction Score Not Applicable No score Not Achieved 0 Partly - Blocking 1 Partly - Non Blocking 2 Fully Achieved 3 Figure 2: OI Step Satisfaction Scoring 2.3 OI Step Maturity Assessment per Validation Phase The OI step maturity is assessed per validation phase. The average assessed satisfaction is calculated as shown in Figure 3, and is used to determine if the maturity criteria has been satisfactorily achieved. Average Satisfaction (%) = Assessed Score of all Applicable Criteria Maximum possible score for Applicable Criteria Figure 3: Average Satisfaction Calculation The OI step maturity assessment for a validation phase is declared as Satisfactorily Achieved or Validation still required based on the average satisfaction, as shown in Figure 4. OI Step Assessment Rationale Satisfactorily Achieved Average satisfaction >= 85% Validation still Required Average satisfaction < 85% Figure 4: OI Step Assessment An overall view of OI step maturity assessments may be produced as shown in Figure 5. OI Step V1 Assessment V2 Assessment V3 Assessment Satisfactorily AO-nnnn Achieved Satisfactorily Achieved Validatation Required Satisfactorily TS-nnnn Achieved Validatation Required Figure 5: Overall OI Step Assessment 2.4 Progress towards OFA Maturity The maturity assessment for all OI steps within the OFA provides a measure of the progress towards OFA maturity. 6 of 13

7 Points are allocated for each validation phase of an OI step as shown in Figure 6. M (V1) [V1]: "Satisfactorily Achieved" = 1 pts, "Validation Required" = 0.5pts, No Assessment = 0pt M (V2) [V2]: "Satisfactorily Achieved" = 2 pts, "Validation Required" = 1pts, No Assessment = 0pt M (V3) [V3]: "Satisfactorily Achieved" = 3 pts, "Validation Required" = 1.5pts, No Assessment = 0pt Figure 6: Points allocation The total of number of points gained when compared to the maximum total number of points possible for an OI step (6 points) provides a measure of the OI step maturity (Figure 7). OI Step Points = Maturity = M (V1) + M (V2) + M (V3) OI Step Maturity = Total Assessed OI Step Points / Total max. OI Step Points Figure 7: OI Step Maturity Calculation The total of number of points gained when compared to the maximum total number of points possible for all OI steps within an OFA provides a measure between 0% and 100% of the progress towards OFA maturity (Figure 8). OFA Maturity = Total Assessed OI Step Points in OFA / Total max. OI Step Points in OFA Figure 8: OFA Maturity Calculation Figure 9 provides an example of the progress towards OFA Maturity calculation. In the example, the OFA contains three OI steps where: Total number of points gained in the OFA = 12.5; Maximum Total number of points possible 3 = 18; Resulting progress towards OFA Maturity : 69 %. OIs AO-nnnn CM-nnnn TS-nnnn OFA 0n.0n.0n V1 V2 V3 Total Assessed Maturity Pts Assessed Maturity Pts Assessed Maturity Pts Total Max. Points Points OFA Maturity Satisfactorily Satisfactorily Satisfactorily Achieved 1 Achieved 2 Achieved =(12.5)/18 Satisfactorily Validatation Achieved 1 Required 1 No Assessment % Satisfactorily Satisfactorily Validatation Achieved 1 Achieved 2 Required OFA Maturity = Total Points / Total max. points Figure 9: OFA Maturity Example Calculation 3 Maximum total number of points for one OI step = 6 pts. 7 of 13

8 3 MAT Report Content 3.1 References The MAT report title page provides the reference of the material used to generate the report (Figure 10): Version of the integrated roadmap; Version of the DMT Report; Version of B.05 Performance Assessment. 3.2 OFA Maturity Figure 10: Reference Material (Example) The MAT report provides a summary for each OFA (Figure 11). See OFA Code & Title See OFA Progress See Performance Assessment See OI Step Maturity Summary Figure 11: OFA Summary (Example) 8 of 13

9 3.2.1 OFA Code and Title The OFA code and title are derived from the latest applicable dataset OFA Progress The progress towards OFA maturity consolidates the assessed V1, V2 and V3 SESAR validation maturity criterion for all OI steps within an OFA Performance Assessment The Step 1 Performance Assessment results are intermediate and provided by SESAR B.05 Project 4. This intermediate performance assessment has focused on the benefits that can be achieved for Step 1, which are in addition to the benefits of the Deployment Baseline 5. Therefore, the assessment assumes that the benefits targeted for the Deployment Baseline are achieved independently from SESAR Step 1 Deployment. Due to the different maturities of projects, the data used for the performance assessment are considered to have varying levels of confidence and hence there is potential for degrees of uncertainty in the assessment result. In some OFAs the assessment has been made on the basis of exercise results, whilst in others, pending the availability of exercise results, the assessment is based on OFA coordinator / project team s expert judgements. Confidence in the B.05 performance assessment process is a simple scale of High, Medium, and Low that is applied by B.05 judgement, and confirmed by OFA/project feedback during the OFA assessment analysis. The Step 1 Performance Assessment is performed at OFA according to Data Set 10. Benefits identified at the OFA assessment stage were analysed and aggregated to obtain results at ECAC level or at a level relevant for each Key Performance Area (KPA). The assessment is based on the Primary Project results related to the OFAs. It captures the benefit of the OFA and OI step as demonstrated by the validation exercises. B.05 uses the KPAs and KPIs (Key Performance Indicators) previously agreed between B.05 and the SESAR programme, and as developed by the Performance Framework of B Fuel Efficiency: percentage reduction in fuel burn. The aggregation provides an overall estimation of the benefit ECAC-wide; Airspace Capacity: percentage of additional peak airspace capacity, representing the sustainable increased throughput. This is considered as a capacity increase at already constrained or at-limit volumes of airspace 6 and hence the aggregation is at this local level. Additionally, airspace capacity is considered separately for TMA (Terminal Manoeuvring Area) and en route airspace. 4 B.05-D68 Updated Performance Assessment in The Deployment Baseline was previously known as IP1. 6 The airspace capacity KPI is not impacted when considering low and medium complexity airspace. 9 of 13

10 Airport Runway Capacity: percentage increase in peak airport runway capacity at already BIC (Best in Class) airports (local level), representing the sustainable increased runway throughput; Predictability: reduction in variability of block to block flight execution time compared to the Reference Business Trajectory 7, i.e. the flight plan that is agreed shortly before going off block. This is initially assessed as a variance across each flight phase, with a final aggregation to a standard deviation value. This assessment focuses on ATM-related predictability and hence the turnaround process is not included in the measurement of the KPI (ECAC level); Cost Effectiveness: reduction of direct ANS cost per flight (ECAC level). This has been assessed by B.05 by taking the improvements foreseen by the OFAs in terms of ATCO productivity and translating this benefit into the reduction of ANS direct gate-to-gate cost per flight 8, indicating potential reduction in user ATM charges. SESAR is also expected to impact ANS costs by affecting technology-related costs. However, this has not been assessed in Cycle 1 and Cycle 2 due to lack of information from enablers OI Step Maturity Summary A summary of OI step assessments per validation phase. The OI step assessment for a validation phase is declared as Satisfactorily Achieved or Validation still Required according to the principle applied in Section For Step 1 this can be considered to be the active trajectory. In Step 2 the term RBT is more common. 8 ATCO costs account for approximately 27% of the overall ANS provision cost. Source: PRR % of direct ANS costs come from technology factors, on which SESAR is expected to also give benefits. 10 of 13

11 3.3 OI Step Maturity The MAT report provides a synthesis for each OI step within an OFA (Figure 12). See OI Step Code & Title See Overview of V1, V2 or V3 maturity See Evidence of Validation. See OI Step Spider Chart Figure 12: OI Step Summary (Example) OI Step Code and Title The OI step code and title are derived from the latest applicable dataset Overview of V1, V2 or V3 maturity The overview of V1, V2 or V3 maturity is a measure of the average assessed satisfaction for an OI step. The OI step assessment for a validation phase is declared as Satisfactorily Achieved or Validation still Required according to the principle applied in Section of 13

12 3.3.3 Evidence of Validation A Definition Maturity Target (DMT) view provides: the defined target Release (corresponding to column in greyed cells); the validation exercises, already executed or planned in the future, that are expected to contribute to OI step maturity assessment according to the last possible M9 milestone date of a Release (shown in brackets). If a DMT view is not provided then evidence of validation, already executed or planned in the future, is provided, for example, demonstration activities, validation planned via OFA Description forms, and OI step covered by OSED or VALP OI Step Spider Chart The spider chart provides a measure of the average assessed satisfaction for the SESAR Validation Maturity Criteria threads 10 within the assessed validation phase. An empty spider chart diagram (blue dot) indicates that validation activity may be planned for the OI step in the assessed validation phase, but an assessment by an OFA Coordinator has not yet been provided. The source of the maturity assessment is indicated in the Spider Chart: Ad-hoc: The maturity assessment was provided by either the OFA Coordinator or SJU through ad-hoc analysis of the material available at the time. Rx SE#3: The maturity assessment was confirmed at the indicated Release Systems Engineering Review #3 based on stable material and therefore indicates a higher level of confidence. 10 All SESAR Validation Maturity Criteria are equally weighted when calculating the average assessed satisfaction per thread. 12 of 13

13 -END OF DOCUMENT- 13 of 13