Methodology for Modeling, Simulation, and Analysis Support of DoD Space Acquisitions

Transcription

1 Methodology for Modeling, Simulation, and Analysis Support of DoD Space Acquisitions Aerospace USC Technical Interchange Meeting Dec 17, 2010 Michael Baxter, Director Modeling & Simulation Systems Analysis & Simulation Subdivision Engineering and Technology Group The Aerospace Corporation 2010

2 Outline Role of Modeling, Simulation, and Analysis System Models Purpose of Modeling, Simulation, and Analysis (MS&A) A Generic Process Defining the Problem The Acquisition Driven Modeling Hierarchy Problem Entry Level Space Acquisitions Problem Entry Levels in Acquisition Traditional Approach to MS&A Evolving Approach to MS&A Summary

3 Role of Modeling, Simulation, and Analysis Develop a simplified Representation of some aspect of a System or Object That performs as the actual system does (Model) Organized as relationships between inputs and outputs (Simulation) Such that the Models and/or Simulations predict the performance, response or reaction of the actual system Identify and Evaluate Essential Functions and Metrics (Requirements)

4 System Models Functions tasks or activities performed by a system Encode Data Store Data Modulate Signal Transmit Performance Parameters measures of how well functions are performed C N d od db 10log PG s sd 20log 4 d R 10log G T d d 10 log L u 10log k Architecture components or objects and interfaces Scheduled Tx U/L D/L Unprocessed Data Processed Data User Data & Prioritization Mission Data

5 What MS&A Attempts to Achieve (Purpose) Explore Identify Measures of Value and Utility Look for Alternative Solutions Develop Specify and Validate Relationships between Capability, Functions, Architecture, and Metrics Assess Alternative Solutions in terms of Utility, Effectiveness, and Performance (also feasibility and risk) Establish Minimum and Desired Targets (function and metrics) Acquire Establish Criteria (Requirements) for Evaluation of Issues in meeting the requirements Contractor Proposals Systems during design, build, test, delivery, deployment and operation

6 Working Information Expectations Deficiencies (Not being meet)) Uncertainty in meeting Opportunities (New) Statement of Work Location in Hierarchy Functional Model(s) Environment Evaluation Metrics Generic MS&A Process Define Problem Concept of Functional Operations Elements Functional Model Problem Context Allocation of Architecture to Functions Metrics on Architecture Code & Tools Inputs Scenarios Simulation Tools Data & Checkout Study Products Evaluation of Solutions Risk Reduction (Rgmts) Program Mgmt (T&E) Data & Scenarios Physical Engineering Models Analysis Model/Tool Analyses Fail to Validate Fail to Verify Verification, Validation, & Accreditation Successful VV&A Fail to Accredit

7 Defining the Problem Problem A divergence between an existing or expected state or condition and an envisioned state (or condition) Where achieving the envision state is worth the expenditure of resources Problems within an Acquisition Context (Problem Levels) What effects have value What capabilities can deliver effects How useful (utility) are the capabilities What functions are needed to deliver capability Define Problem How well (effectiveness) can the functions provide capability What architecture can provide functions How well (performance) can the architecture perform the functions What specifications are necessary to build the architecture What characteristics must the architecture possess (technical performance measures TPMs) to achieve the performance Which of the above need to be required (requirements) Fail to Accredit

8 Acquisition Driven Modeling Hierarchy What example Metric example Effect Deter Attack Value No Attack: Good Attack: Bad Global Strike Force Exchange Capability Utility Ratio Function Transport Weapons Measure of of of Effectiveness Payload to Target Architecture Orbital Vehicle Performance Measures Range Specification Titan IV Technical Performance Measures Weight

9 Problem Entry Level Effect Value What effects have value? Capability Utility What capabilities can deliver effects How useful (utility) are the capabilities Function Architecture Measure of of of Effectiveness Performance Measures What functions are needed to deliver capability How well (effectiveness) can the functions provide capability What architecture can provide functions How well (performance) can the architecture perform the functions Specification Technical Performance Measures What characteristics (specifications) must the architecture possess (TPMs) to achieve above metrics Which of the above need to be required (requirements)

10 Acquisition Phases Tailored Process for Space Draft USECAF Space Acq Policy Small Quantity Acquisition Phase Model -- NSS Acq Policy Pre-Systems Acquisition Systems Acquisition Sustainment Key Decision Points: JROC ICD PHASE A Approval A JROC CDD PHASE B Approval B JROC CPD PHASE C Approval C Follow On Buy Approval 1st Launch IOC Upgrade Decision FOC Pre KDP-A Activities PHASE A (Study Phase) Concept/ Architecture Dev PHASE B (Design Phase) Risk Reduction & Design Development PHASE C (Build, Test, Launch) Acquisition & Operations Support Concept Decision Mtg SRR SDR PDR CDR

11 Effects Value Capabilities Utility Functions Effectiveness Capabilities Utility Functions Effectiveness System Architecture Performance Functions Effectiveness System Architecture Performance Design Specs TPMs Functions Effectiveness System Architecture Performance Design Specs TPMs Modeling and Simulation Problem Entry Levels in Acquisition Pre KDP A Phase A Phase B Phase C Concept Concept & Risk Reduction Acquisition Exploration Architecture & Design Dev. & Ops. Support Dev. Mission Family of Systems System of Systems System Subsystem Components TPM Technical Performance Measure The Acquisition Phases lay out the level of model/metric detail at which solution are to be developed

12 Traditional Approach to MS&A Motivation for Systems Development and Acquisition Replacing Aging Systems (Opportunity) Technology Developments or Demonstrations (Opportunity) Commanders Identify Issues in Dealing with Threats (Deficiency) Cost or Budget Issues (Deficiency) Generally Metrics Identified in the Use of Existing Systems Replacing or Improving an Existing Capability Program Office Generally Provided Metrics to Be Evaluated Problem Entry Level (informally as assumptions and constraints)

13 Evolving Approach to MS&A Going to a Capability Based Approach Envisioning New Effects and Capabilities Looking for Synergy between Capabilities Top-Down Driven Captured as Architecture Frameworks Analysis Results Dependent on Un-Answered Questions at Higher Problem Levels Architecture Frameworks Address the Architecture Values and Utility often untested assumptions Metric Selection, Decomposition, and Evaluation often left to the analyst Capability Based Approach Challenges the MS&A Analyst to Get in Early before Architecture Decisions Lock in Utility, Effectiveness and Performance Push Back on Policy Makers Be Able to Work the Proceeding Problem Levels

14 Summary Validity of MS&A is Dependent on Answers at Higher Problem Levels Customers Role in Defining the Metrics and Metric Answers Decreasing Defining Capabilities and Architecture Increasing Potential for Serious Disconnects Role of Risk Management Uncertainty in Capabilities, Functions and Metrics Awareness of Problem Levels Key to Working the Right Questions Identifying Useful Metrics Assessing the Range of Alternative Solutions Developing Verified, Validated and Accredited MS&A Tools The Role of MS&A in Driving Architecture Selection Through Evaluation of Relevant Metrics Moving from Customer to Analyst Not Always Appreciated by the Customer

15 Additional Material

16 How are Requirements Used As a minimal and lowest detail bounds on The set of functions and performance measures and targets That provides assurance that the proposed solution systems will Be able to satisfy the users needs by achieving the operational requirements In the expected and likely environments Yet minimizes selection of a specific architecture To allow for innovative and new architecture/technology approaches Requirements (documents) Proposed Architecture 1 Function 2 Function 1 Function 12 Functional Function 9 Function 2 Function 1 Function 3 Function 5 Functional Function 4 System Ground Seg. Space Seg. Mobile Seg. MoE 3 MoE 2 = funct (TPM 5) = funct (TPM 4) TPM 4 Risk MoE 3 MoE 2 = funct (TPM 4) = funct (TPM 2,TPM 3) = funct (TPM 1) MoE 1 TPM 1 TPM 4 TPM 2 TPM 3 Proposed Architecture 2 Function 2 Function 1 Function 12 Functional Function 9 Proposed Architecture that Most fully meets the Functional Requirements with the Best Performance and Lowest Risk = funct (TPM 1) MoE 1 Government Performance Requirements MoE 3 MoE 2 = funct (TPM 4) = funct (TPM 5) TPM 4 TPM 5 = funct (TPM 1) MoE 1 TPM 1 Proposed Architecture 3...

17 Program Phases of Requirements Development Joint Concepts Integrated Architectures (JCIA) "What" "How well" Functional STAR TEMP ICD CONOPS TRD CDD CPD Gov't (users/ operators) Gov't (acquisition) Physical S eg X s pec Seg Y spec Seg-Z spec Contractor "How" Sub-X1 spec Sub-Y1 spec Sub-Z1 spec

18 Functional Modeling Concept of Operations Functional Elements Problem Context Concept of Operations Functional Model The capabilities and functions Fail to Validate envisioned for conducting mission operations and their relationships to potential architecture, the environment, and resources. DoD: A verbal or graphic statement, in broad outline, of a commander's assumptions or intent in regard to an operation or series of operations. Functional Elements The functions (activities, actions, operations, etc.) to provide Capability including Sequence Options under different conditions (Modes) Problem Context Store Data Timeline Encode Data Interfacing Systems Modulate Signal Environments and Resources Scenarios Concepts of Operations Transmit

19 Physical Engineering Models Architectural (Objects and Interfaces) Objects (Software, Hardware, Procedures, ) provide/accomplish functions Functions are selected via allocation to Architecture Notional Architectures used for Exploration Mathematical Value Scales Utility Measures Effectiveness Measures Performance Measures Technical Performance Measures Behavioral Rules of Engagement/Interaction Decision Trees/Probabilities Logical Statements, C N d od 10log PG User Data & Prioritization 4 R 20log s sd db d if x >.5 then y, else z U/L D/L Scheduled Tx G 10log T d d Physical Engineering Models Unprocessed Data 10log L u Processed Data Mission Data 10log k

20 Analysis Models and Tools Assembling Model Based Tools Entry Level based on what questions are being asked Data & (or which answers are uncertain) Scenarios Begin with the Metrics that make a difference Determine the range of Capabilities/Functions/Architecture Determine which parameters/properties determine the metrics Select a useful medium in which to model and simulate Develop the relationships (guided by the analysis objectives) Scenarios Input values and Initial Conditions Architectures and Objects and their properties Rules of Interaction/Engagement Antenna Gains Mission/Operations Objectives Data Input Properties and parameters which determine the values of the metrics Scenarios Output Metric values Sequences, interactions, state changes Red Force consists of Tanks Probability of Detection Analysis Model/Tool Fail to Verify

21 Verification, Validation & Accreditation Data & Scenarios Verification, Validation, & Accreditation Fail to Verify Fail to Validate Fail to Accredit Verification Substantiating that the model is transformed from one form into another, as intended, with sufficient accuracy. Model verification deals with building the model right. Validation Substantiating that the model, within its domain of applicability, behaves with satisfactory accuracy consistent with the MS&A objectives. Model validation deals with building the right model. Accreditation the official certification that a model or simulation is acceptable for use for a specific purpose. DoD Directive

22 Analysis Data & Scenarios Successful VV&A Analysis of Systems Breaking a system into components and properties Determining the relationships between components and properties Modeling the significant relationships Analyses Assembling the significant relationships to model the system properties and relationships Using such models to estimate the states and transitions in states given initial conditions, inputs, and scenarios Comparing the desired values, utility, effectiveness, performance, and specifications against the modeled states and state transitions

23 Basic Definitions System collection of entities, acting/interacting together to achieve some logical end (purpose). Mathematical Model a representation of a system in terms of logical and quantitative relationships that are then manipulated and changed to see how the model reacts, and thus how the real system would react if the mathematical model is a valid one. Simulation a model organized into inputs, outputs, and relationships between inputs and outputs Static: Time plays no role. Dynamic: Evolves over time Discrete Event: Varies across separate points in time Continuous: Varies continuously over time Deterministic: No probabilities Stochastic: Probabilities Reference: Law, A. M. and Kelton, W. D., Simulation Modeling & Analysis, McGraw-Hill, United States of America, 1991.

24 JCIDS and System Engineering Strategic Policy Guidance Systems Engineering Joint Operations Concepts Joint Operating Concepts Joint Functional Concepts Integrated Architectures Functional Area Analysis Functional Needs Analysis Functional Solution Analysis Trade Space Exploration Notional Systems Modeling Desired Operational Functions and Effects Test & Evaluation Planning Risk CAIV Technical Requirements Generation (TRD) TRD CPD CDD ICD Acquisition & Program Management

25 What Are Requirements Capability based requirements are bounds On how the users needs will be achieved In terms of The capabilities or functions a system will perform The performance of those functions (and the associated operating environments) The minimal organization of those functions (Segments etc.) The existing/planned external interfaces The Other Set of Requirements is Design requirements Or a description of the architectural components and interfaces Left open (as much as possible) to maximize innovation Examples Capability: provide battlefield situational awarness Function: transmit IFF(ID friend or foe) data to battlefield Performance: Bit Error Rate (BER) < 1/ Design: 24 Gigabyte Microprocessor Functional Requirements Analysis Function 1 Functional Function 2 Function 3 Function 4 Function 5 Notional Architecture(s) Performance Requirements Analysis MoE 3 MoE 2 Scheduled Tx User Data & Prioritization = funct (TPM 1) U/L = funct (TPM 4) = funct (TPM 2,TPM 3) MoE 1 TPM 1 D/L TPM 4 Unprocessed Data Processed Data Mission Data TPM 2 TPM 3 Measures of Effectiveness Technical Performance Measures

26 References CJCSI C Joint Capabilities Integration and Development System, 24 Jun 2003 CJCSM M Operation of the Joint Capabilities Integration and Development System, 24 Jun 2003 DODI The Defense Acquisition System, 12 May 2003 DODD Operation of the Defense Acquisition System, 12 May 2003 Defense Acquisition Guidebook DoDI ; Implementation of the Planning, Programming, and Budgeting System (PPBS); Thru Change 1; April 9, 1987 DoDD ; The Planning, Programming, and Budgeting System (PPBS); (Including Change 1); 28 July 1990 NSS 03-01National Security Space Acquisition Policy, 6 Oct 2003