An Enhanced Analysis of Alternatives (AoA)

Transcription

1 An Enhanced Analysis of Alternatives (AoA) A Mission-Oriented, Evaluation-Based Framework for Defense Test Evaluation NDIA March Vince Roske Institute for Defense Analyses vroske@ida.org

2 Role of the Analysis of Alternatives (AoA) Analysis of Alternatives (AoA) Translates tasks to be performed in an operational context into a system solution for acquisition Strategic Guidance Joint Concepts CBA JCD MDD Capability Need MS A Materiel Solution Tech Dev Analysis MS B Engineering Manufacturing Development Demonstration MS C CPD Production Deployment OSD/JCS COCOM FCB Task performance Operational context AoA KPPs KSAs Solution description Incremental Development Acquisition Strategy OS JCIDS definition of AoA: The evaluation of the performance, operational effectiveness, operational suitability, and estimated costs of alternative systems to meet a mission capability..the AoA is one of the key inputs to defining the system capabilities in the Capability Development Document () Insights Needed to Build a TEMP 2

3 AoA Methodology Strategic Guidance OSD/JCS Joint Concepts COCOM CBA FAA FNA JCD FCB AoA Performed by a JROC Designated (Service) Sponsor MS A MS B Materiel Tech Dev MDD Solution Analysis AoA Engineering Manufacturing Development Demonstration EVALUATION FRAMEWORK SCENARIO (s) MS C CPD Incremental Development? CBA TASKS CONDITIONS CAPABILITY NEED Full Rate Prod DR Production Deployment OS Solution Alternatives Force Structure CONOP OPTEMPO Mission Performance Solution Descriptions Performance Characteristics Development Testing Acquisition Alternatives Procurement Specifications OPERATIONAL CONTEXT MS B 3

4 A Review of AoAs Reviewed AoAs AoAs Done Between 2003 and 2008 AND Done for DOTE Oversight List Programs Implications for TE: Each Contains a Description of an Operational Context Potentially Useful to TE Planning Scenario, Forces, Objectives CONOPS, Climate, OPTEMPO, etc Each includes a Sensitivity Analysis (SA) Relating System Performance to Mission Accomplishment HOWEVER: SAs do NOT Relate Variations in System Characteristics Performance to Mission Effectiveness SAs Treat Performance Characteristics as Fixed ; Used to Derive System Fleet Size or System Configurations of Fixed Components for Mission Accomplishment Across Varied Scenarios AoA Informs a System Acquisition Strategy» A Packing Algorithm Paradigm: How Many Systems as Defined are Needed, When, for Mission Success and at What Cost AoA Does NOT Recognize TE as a Customer» TE Examines System and Component Characteristics Performance relative to Mission Effectiveness measures 4

5 Analysis of Alternatives (AoA) Requirements-Based AoA EVALUATION FRAMEWORK Fixed Performance Characteristics VARY Fleet Size, Mix, or System Composition To Operate in the Force Structure -DOTLPF, CONOP, OPTEMPO To Accomplish the Specified Missions SCENARIO (s) Across the Scenarios Force Structure CONOP OPTEMPO Mission Performance Metrics OPERATIONAL CONTEXT CAPABILITY NEED Tasks Conditions Solution Alternatives -Physical Description -Suitability Characteristics -System Performance SENSITIVITY ANALYSIS Solution Descriptions Acquisition -Fleet Size Cost -Procurement Schedule MS B AoA Performed by JROC Designated (Service) Sponsor 5

6 An Enhanced Analysis of Alternatives (AoA) Capability-Based Sensitivity Analysis EVALUATION FRAMEWORK VARY the System Characteristics In the Sensitivity Analysis -Develops the Performance, Effectiveness, Cost Trade Space To Operate in the Force Structure -DOTLPF, CONOP, OPTEMPO To Accomplish the Specified Tasks Across the Scenarios Conditions Force Structure CONOP OPTEMPO AoA SCENARIO (s) Mission Performance Metrics JCIDS CAPABILITY NEED Tasks Conditions OPERATIONAL CONTEXT Solution Alternatives System Type/Form VARY Characteristics -Physical -Performance AoA Performed by JROC Designated (Service) Sponsor SENSITIVITY ANALYSIS Solution Descriptions MS B System Description -Mission Effective System Characteristics Performance Requirements -Context Conditions for Testing -Development RFP Requirements and Grading Criteria (Sections L M) Acquisition -Fleet Size Cost -Procurement Schedule 6

7 Enhanced AoA Treats System Performance Characteristics as Variables in the AoA / SA Enhanced AoA Sensitivity Analysis (SA) SYSTEM Variable Characteristics Performance Characteristics Fixed / Variable SENARIO 4 SENARIO 3 SENARIO 2 SENARIO 1 Suitability Mission Effectiveness System Fleet Procurement Alternatives and Costs Acquisition Strategy Evaluation Framework Pre-Milestone A AoA Assumptions and Representations for Early Testing Performance Characteristics and Associated Mission Effectiveness and Suitability Conditions for System Testing TEMP Test Results 7

8 An Enhanced AoA an Integrating Process SUITABILITY (Rqmts Evaluation Framework) RAM Compatibility Fuel Energy (FBCF) System Alternatives Interoperability Early Testing of Assumptions Representations (MS) Pre Milestone A AoA Effectiveness Suitability Performance Characteristics Conditions Force Structure CONOP OPTEMPO TEMP SCENARIO (s) MISSION EFFECTIVE OPERATIONAL CONTEXT Early Testing Assumption and MS CAPABILITY NEED TASKS CONDITIONS Operationally Oriented DT Integrated OT JOC-T Training Environments JMETC Testing in Joint Environments (TIJE, CTM) Procurements Alternatives Acquisition Strategy RFP Content Sections LM 8

9 TE Benefits from an Enhanced AoA Enhanced Description of the Operational Context For use in the AoA For deriving operational environments for testing For integrating development and operational test planning Consistent Validated Evaluation Framework For systems design, testing, and assessment of system and mission cost / effectiveness MDA better informed for Confidence in the System and for mitigating any performance deficiencies More Comprehensive and Effective Test Planning Enhancing Confidence in the AoA Early testing of key AoA assumptions and representations Responsive to pre-milestone A program reviews Tasking and Resource Planning in the TEMP» For MS in the AoA» For MS supporting DT and OT events Mission Oriented, Evaluation Based Test Planning Testing integrated with the AoA Testing focused on most important performance characteristics and conditions Test results evaluated for mission effectiveness in the AoA analysis framework Harmonized Acquisition Milestone Decision Support System Design, Cost / Effectiveness Evaluation, Acquisition Strategy, and Test Program 9

10 Benefits to TE: Capability Test Method (CTM) System Design ACQ AoAP Approved AoA Plan AB Analytical Baseline (DPS, MSFD, FYAB) UJTL UJTL JOpsC JOpsC Family (JOC, JFC, JIC) Exploratory Analysis Results JCA Joint Capability Areas CTM.0 Develop TE Strategy Who and how? Fair to TE or others? A parallel to acquisition process? Performance requirements? Evaluation framework? How to synchronize? OTP JCD Other Test Plans (DT, OT,LF) TES Test Evaluation Strategy TEMP TEMP TESTEMP.JOCT Joint Operational Context for Test (JOC-T) Joint Capabilities Document (JCD) CTM.1 Characterize Test Initial Capabilities Document () DCR DOTMLPF Change Request (DCR) PID Program Introduction (PID) SOC Statement of Capability (SOC) Capability Development. Document () CTM.2 Plan Test CPD EJFM Capability Production Document (CPD) Enterprise JME LVC Foundation Model Test Plan 10

11 Benefits to TE: Enhanced CTM Acquisition Strategy JCA Initial Capabilities Document () Joint Capability Areas ENHANCED AoA Process Mission Context Eval Framework Early Testing Develop TEMP Mission Oriented Evaluation Based AoA Production / System Description AoA Related Early Testing MSA TEMP Development Harmony MSB CCD OTP JCD Other Test Plans (DT, OT,LF) Joint Capabilities Document (JCD) Exploratory Analysis Results AB Analytical Baseline (DPS, MSFD, FYAB) UJTL UJTL Joint Operational Context for Test (JOC-T) TEMP Part III TE Strategy New TEMP Format System Evaluation Strategy E Test Strategy E DT,LFT, OT Eval Test Approaches DCR DOTMLPF Change Request (DCR) JOpsC CPD JOpsC Family (JOC, JFC, JIC) Characterize The Tests Capability Development. Document () Capability Production Document (CPD) PID Program Introduction (PID) SOC Statement of Capability (SOC) EJFM Enterprise JME LVC Foundation Model Plan Test Test Plan 11

12 Part I Introduction Benefits to TE: An Emerging TEMP Format Part II Test Program Management Schedule Part III Test Evaluation Strategy Part IV Resource Summary Brief Purpose of TEMP Brief Mission Description Brief System Description -System Threat Assessment -Program Background -Key capabilities -Key Interfaces -Cert Rqmts -Sys Engr Rqmts AoA Overview (TE Perspective) -Operational Context of Evaluations -AoA System/Mission Evaluation Approach Acquisition Strategy Overview -Milestone Information Needs -System Maturity Schedule Describing a Mission-Oriented, Evaluation-Based Test Program TE Management Comment TE Data Base Rqmts Deficiency Reporting TEMP Updates Integrated Test Program Schedule System Evaluation Strategy (TE Perspective, Relates to AoA, Explains TE Testing s Strategy Role in System/Mission AoA Effectiveness Overview Evaluation) -Operational- Joint Operational Context Context for Evaluations -Evaluation- Evaluation Framework Approach -Acquisition -Mission Strategy Effectiveness Evaluations - System -Comparison Maturation to Current Capability - Capabilities -Test Support Limitations to Evaluations Mission Test Strategy Oriented (Testing s Evaluation Approach Framework to Supporting -Joint Operational the Evaluation Context Strategy) for Test -Joint (JOC-T) Operational Context for Test (JOC-T) -Evaluation -DT OT Objectives Approach from Evaluation Strategy -Statutory Evals -Assumptions -Mission and Capability MS Validation Comparison Tests Test -Integrated Strate DT OT Planning Objectives Developmental Evaluation Approach -Developmental Test Approach -MS, Test Limitations Live Fire Evaluation Approach -Test Objectives, MS, Test Limitations Certification for IOTE Operational Evaluation Approach -Operational Test Approach -MS, Test Limitations Other Certifications Reliability Growth Future TE Introduction -Test Articles -Test Sites Instrumentation -Test Support Equipment -Threat Representations -Test Targets Expendables -OPFOR Test Support -MS and Test-beds -Joint Operational Test Environment -Special Requirements Federal, state, Local Rqmts Manpower/ Personnel Training Test Funding Summary 12