Development of a Composite Program Assessment Score (CPAS) for Advanced Technology Portfolio Prioritization

Transcription

1 Development of a Composite Program Assessment Score (CPAS) for Advanced Technology Portfolio Prioritization By Varun Sharma Thesis Co-Advisors: Dr. James T. Luxhøj and Dr. David W. Coit Thesis Proposal Presentation December 16, 2005

2 Presentation Outline Background Aviation AvSSP Program Research Problem Literature Review Methodology Metric Weights Determination Weighted Sum Model Additive Value Theory Model Preliminary Results Part 121/135 Prototype Decision Support Tool Future Work 12/16/2005 2

3 Background - History Father of Aviation - George Cayley ( ) First Piloted Flight - Wright Brothers (December 17, 1903) Commercial Aviation Generates $ 300 billion annually More than 60 new airlines started service since 1992 Millions of customer fly every year 12/16/2005 3

4 Background Aviation Safety Aviation is one of the safest mode of transportation It is 22 times safer to fly than travel by car (US National Safety Council, ) Increasing air traffic is an area of concern U.S Transportation Accident Rate Fatalities History /16/2005 4

5 Background - AvSSP 1996 White House Commission on Aviation Safety and Security NASA/FAA created the Aviation Safety Program (AvSP) Strategic Plan Reduce Aircraft Fatal Accident Rate by 80% within 10 years and by 90% within 25 years Post 9/11 renamed Aviation Safety and Security Program (AvSSP) 12/16/2005 5

6 Background - AvSSP Safety Program Avert unintentional life threatening events Develop prevention, intervention and mitigation technologies and strategies High priority to factors considered largest contributor to accident and fatality rate 12/16/2005 6

7 Background Safety Product Suite Focus Aircraft Self-Protection and Preservation Environmental Hazards Awareness and Mitigation System Vulnerability Discovery and Management Vehicle Safety Technologies Product Category Weather Safety Technologies System Safety Technologies Synthetic Vision Systems (SVS) Single Aircraft Accident Prevention (SAAP) Accident Mitigation (AM) Technology/Intervention Suites Weather Accident Prevention (WxAP) Aircraft Icing (AI) Aircraft System Monitoring and Modeling (ASMM) System Wide Accident Prevention (SWAP) 12/16/2005 7

8 Background Safety Program Assessment Safety Metrics: Technical Development Risk Implementation Risk Fatal Accident Rate Safety and Cost Benefits Safety Risk 12/16/2005 8

9 Background Safety Program Assessment Metrics Metric: Technical Development Risk Source : NASA, FAA, Volpe, Sandia, The Aerospace Corporation, Georgia Institute of Technology and University of Virginia Objective : Considers required technical advancement, technology status, complexity, dependencies, testability/verifiability and impact on technology goal Goal Limits : 0 to 1 : Minimization Metric: Implementation Risk Source : NASA Langley/NASA Glenn/Swales Aerospace Objective : Considers certification process, dependencies, market penetration and market impacts Goal Limits : 0 to 1 Metric: Fatal Accident Rate Source : Minimization : DAI Technologies/NASA Langley/NASA Glenn Objective : % decrease in fatal accident rate - direct and indirect impacts Goal : Maximization Limits 12/16/2005 : 0% to 100% 9

10 Background Safety Program Assessment Metrics Metric: Safety and Cost benefits Source : Volpe National Transportation Center Objective : Capture the safety benefits (i.e., reductions in injuries, Goal Units fatalities, hull loss, and other accident outcomes) and business benefits (avoiding costs of cancellations, delays, litigations, regulatory actions, lost market valuation) : Maximization : $/flight-cycle saving Metric: Safety Risk Source : Rutgers University Objective : Shows the projected impact that AvSP products may have on reducing aviation risk in National Airspace System (NAS) Goal : Maximization Units : 0% to 100% 12/16/

11 Research Problem Five program assessment metrics provide an objective measurement tool Customer portfolio suggests conflicting objectives Development of a unified metric that simultaneously considers each metric 12/16/

12 Research Objective 1 Development of an Analytic Framework/Structure of Composite Program Assessment Score (CPAS) Identify the multi-criteria decision making methodology applicable to the problem at hand Develop a defensible/understandable analytical method for aggregating five conflicting metrics to a composite score 12/16/

13 Research Objective 2 Demonstration of Analytical Methods with data Utilize Subject Matter Experts (SMEs) knowledge to obtain results using methodologies identified Compare the results based on different methodologies used 12/16/

14 Research Objective 3 Development of a Prototype Decision Support Tool Develop a prototype tool with a user interface based on the analytical models built Tool should require minimal input and provide understandable and repeatable output Tool should relieve user from tedious calculations 12/16/

15 Literature Review Multi-Criteria Decision Making Methodologies: Weighted Sum Model Multi-Attribute Utility Theory Analytic Hierarchy Process (AHP) Goal Programming Pareto Optimality Multi-Attribute Value Theory 12/16/

16 Literature Review: Summary Method Advantages Limitations Weighted Sum Model Multi-Attribute Value Theory Easy to understand Valid approach Easy to implement Requires fewer inputs from the user Repeatable Considers non-linear behavior of the metrics Accurate Repeatable Assumes Linearity Assumes Independence Additive model assumes independence Initial model building requires more inputs from user 12/16/

17 Methodology Metric Weight Determination Weighted Sum Model Additive Value Theory Model 12/16/

18 Methodology: Weight Determination Metrics Weight Determination Metric weight represents relative importance One set represents a scenario Methodology draws analogy from AHP Step-wise procedure Select most important metric Assign scores using weight assignment scale Calculate weights Score 0 Weight Assignment Scale Definition Absolute Weakness / No Importance 2 Very Strong Weakness 4 Strong Weakness 6 Medium Weakness 8 Slight Weakness 10 1,3,5,7,9 Equal Importance / No Weakness Intermediate values between adjacent judgments 12/16/

19 Methodology: Weight Determination Program Assessment Metric Score Weight Technical Development Risk Implementation Risk Fatal Accident Rate Safety and Cost Benefits Safety Risk 10 10/28 = 0.36 Sum = 28 12/16/

20 Methodology: Weighted Sum Model Identification of orientation of the metrics Scaling of metrics Common orientation Dimensionless scale f i = f high f high metric f low i 12/16/ i metric high metric i is to be maximized metric i is to be minimized f high /f low selection Mathematical Limit Highest/Lowest value in data set f = f i f f low low

21 Methodology: Weighted Sum Model Weight determination CPAS calculation INPUT Program assessment scores for individual metrics from NASA 1. Identification of Metrics Orientation Use metric definitions to identify metric orientation 2. Scaling Determine scaled values using formulas CPAS= where, i= i= 1 w w i i f i 3. Weight Determination Determine weights using relative scores 4. CPAS Calculation Calculate CPAS using formula OUTPUT Objective evaluation of Technologies/Interventions WSM Method 12/16/

22 Methodology: Additive Value Theory Model Identification of orientation of the metrics Representative value function determination v i /x i values Function value v i (x low ) v i (x low/med ) v i (x med ) Metrics to be maximized x low = lowest value in data set / mathematically possible X low/mid = mid-value point of [x low,x mid ] x mid = mid-value point of [x low,x high ] Metrics to be minimized x low = highest value in data set / mathematically possible X low/mid = mid-value point of [x low,x mid ] x mid = mid-value point of [x low,x high ] Regression modeling v i (x med/high ) X mid/high = mid-value point of [x mid,x high ] X mid/high = mid-value point of [x mid,x high ] v i (x high ) x high = lowest value in data set / mathematically possible x high = highest value in data set / mathematically possible 12/16/

23 Methodology: Additive Value Theory Model Scaling of metrics Common orientation Dimensionless scale Uses value functions determined previously Weight determination CPAS calculation CPAS= where, i= i= 1 w w v i i i INPUT Program assessment scores for individual metrics from NASA 1. Identification of Metrics Orientation Use metric definitions to identify metric orientation 2. Representative Value Function Determination Subject Matter Expert input for determining representative value function 3. Scaling Determine scaled values using function identified 4. Weight Determination Determine weights using relative scores 5. CPAS Calculation Calculate CPAS using formula OUTPUT Objective evaluation of Technologies/Interventions Additive Value Theory Method 12/16/

24 Results: Weighted Sum Model Representative scenario Common Airline Development Agency Government Agency Common Airline with recent history of problems 12/16/

25 Results: Weighted Sum Model Common Airline Scenario Common US airline Operates both Part 121/Part 135 aircrafts No major problems in recent history Recent government regulation to implement these technologies/interventions in next five year 12/16/

26 Results: Weighted Sum Model Common US Airline Scenario User Profile 0.40 A common USA airline Operates 0.30 both Part 121 and Part 135 aircrafts 0.20 No major problems in recent 0.10 history Government 0.00 has passed a regulation to implement these technologies/interventions in the next five years Part 121/135 CPAS Program Assessment Metric Technical Development Risk Implementation Risk Fatal Accident Rate Safety and Cost Benefits Safety Risk Score Weight AM - 1 AM - 4 AM - 6 ASMM - 1 ASMM - 3 ASMM - 5 SAAP SAAP - 2 (Propulsion) SAAP SAAP - 6 (Neural Network Validation) SAAP - 7 (Fault Tolerant Modular Archirecture) SAAP - 8 SVS - 1 SVS - 4 SWAP - 1 SWAP - 3 SWAP - 5 SWAP - 7 SWAP - 9 WxAP - 1 & 3 WxAP - 5 WxAP - 7 AI - 2 AI - 4 AI - 6 AvSP Product 12/16/ CPAS

27 Results: Weighted Sum Model Development Agency Scenario Responsible for coordinating development of technologies/interventions Development guidelines Official target to reduce fatalities in aircraft accidents Highly influential target industry 12/16/

28 Results: Weighted Sum Model Development Agency Scenario User Profile Responsible for coordinating development of technologies/interventions Development guidelines Official target to reduce fatalities in aircraft accidents Highly influential target industry Part 121/135 CPAS Program Assessment Metric Technical Development Risk Implementation Risk Fatal Accident Rate Safety and Cost Benefits Safety Risk Score Weight 12/16/ AM - 1 AM - 4 AM - 6 ASMM - 1 ASMM - 3 ASMM - 5 SAAP SAAP - 2 (Propulsion) SAAP SAAP - 6 (Neural Network Validation) SAAP - 7 (Fault Tolerant Modular Archirecture) SAAP - 8 SVS - 1 SVS - 4 SWAP - 1 SWAP - 3 SWAP - 5 SWAP - 7 SWAP - 9 WxAP - 1 & 3 WxAP - 5 WxAP - 7 AI - 2 AI - 4 AI - 6 AvSP Product CPAS

29 Results: Additive Value Theory Model Non-linear Model Session with Aviation Safety Expert Determination of midpoint values Function Value (v i ) Technical Development Risk (m 1 ) Implementation Risk (m 2 ) Fatal Accident Rate (m 3 ) Safety and Cost Benefits (m 4 ) Safety Risk (m 5 ) v i (x low ) = 0.00 x low = 0.74 x low = 0.85 x low = 0.00 x low = 1.00 x low = 0.0 v i (x low/med ) = 0.25 x low/med = 0.67 x low/med = 0.59 x low/med = x low/med = x low/med = 3.6 v i (x med ) = 0.50 x med = 0.55 x med = 0.40 x med = x med = x med = 5.6 v i (x med/high ) = 0.75 x med/high = 0.38 x med/high = 0.21 x med/high = x med/high = x med/high = 7.2 v i (x high ) = 1.00 x high = 0.14 x high = 0.10 x high = x high = 234 x high = /16/

30 Results: Additive Value Theory Model Regression Modeling Comparison of functions based on R- square values Quadratic function best fits the data Further analysis to be done Value Technical Development Comparison Metic Value 2 y = x x Y value V1 (x) = Linear V2 (x) = Quadratic V3 (x) = Exponential V4 (x) = Power 12/16/

31 CPAS Decision Support Tool CPAS DST is a prototype tool for performing CPAS analysis Uses Microsoft Excel and embedded Visual Basic features Provides rank ordered technologies/interventions Compares scenarios Filters as per accident type 12/16/

32 CPAS Decision Support Tool Common US Airline Initially Safety and Cost Benefits Metric is assigned a score of 10 Assign scores to remaining metrics using scale for weight assignment This generates the scenario with computed weights Left click on Productwise CPAS to go to next screen 12/16/

33 CPAS Decision Support Tool Output screen shows individual technologies/interventions with respective CPAS Left click on Product wise CPAS (ordered) to go to next screen This generates Output screen with ranked technologies/interventions with respect to CPAS 12/16/

34 CPAS Decision Support Tool Other features Scenario Summary Filters by accident type 12/16/

35 Remaining Work Task 1: Completion of the Additive Value Theory Model Complete scaling and determine weights Complete analysis Conduct one-to-one comparison with Weighted Sum Model for each scenario Task 2: Weight elicitation through sessions with Subject Matter Experts (SMEs) Conduct more sessions for weight elicitation Obtain expert assessment 12/16/

36 Remaining Work Task 3: Analysis under Additional Constraints Consider analysis under additional constraints Task 4: Completion and Refinement of CPAS Decision Support Tool Develop tool for Additive Utility Theory Model Combine Weighted Sum and Additive Utility Theory Models in a single tool 12/16/

37 Thanks to Dr. Luxhøj and Dr. Coit Dr. Boucher NASA Mr. Del Green CARDA Team 12/16/