The Developmental Evaluation. Experiments (DOE) - Better Informed Decisions

Size: px

Start display at page:

Download "The Developmental Evaluation. Experiments (DOE) - Better Informed Decisions"

Curtis Pitts
6 years ago
Views:

Approved for Public Release; Distribution Unlimited.

Design of Experiments (DOE) - Helping Decision i

1 Approved for Public Release; Distribution Unlimited. Case Number The MITRE Corporation. ALL RIGHTS RESERVED. The Developmental Evaluation Framework (DEF) and Design of Experiments (DOE) - Helping Decision i Makers make Better Informed Decisions 3 March 2016 Luis A. Cortes, Ph.D., P.E. The MITRE Corporation lcortes@mitre.org Suzanne M. Beers, Ph.D. DASD(DT&E) DEF Technical Lead The MITRE Corporation sbeers@mitre.org

2 Discussion Topics Context - DT&E s Purpose is to Inform Decisions First the E : Developmental Evaluation Framework (DEF) Then the T : Design and Analysis of Experiments Then the T : Design and Analysis of Experiments (DOE/STAT)

Improving Acquisition Decision- Making Honorable Frank Kendall (USD/AT&L) on the

Acquisition Executive, I rely heavily on the implications of developmental test

Developmental testing is a core activity in our acquisition programs.

so that good decisions can be made as early as possible.

are structured to extract meaningful information as efficiently as possible and I

The bottom line: Developmental testers are critical professionals who make a major

3 Improving Acquisition Decision- Making Honorable Frank Kendall (USD/AT&L) on the importance & benefits of DT&E-informed acquisition decision-making As Defense Acquisition Executive, I rely heavily on the implications of developmental test results for investment decisions,,particularly for entry into low rate production. Developmental testing is a core activity in our acquisition programs. The purpose of developmental testing is simple: to provide data to program leadership so that good decisions can be made as early as possible. Formal design of experiments techniques are being used widely now to ensure that tests are structured to extract meaningful information as efficiently as possible and I applaud this development. The bottom line: Developmental testers are critical professionals who make a major contribution to DoD s programs. Working with program and engineering leadership as key members of the management team, developmental testers provide the information that makes program success possible and much more probable.

4 Attributes of a Good Test A clear problem statement and well understood objectives Plan Suitable response variables to be analyzed An adequate factor space (factors that influence the response) A test design structure that fits the problem Design Test Analyze An adequate test execution strategy Solid statistical data analysis Valid conclusions and practical recommendations The attributes of Good Tests are rooted in the scientific method Montgomery, D. C. (2013), Design and Analysis of Experiments,8 th ed., John Wiley & Sons.

5 DEF & DOE Contributions to a Good Test Plan Execute Data Inform Decisions Measures Factor Space Test Design Test Execution Data Analysis Evaluation and Assessment Developmental Evaluation Framework (DEF) Design and Analysis of Experiments (DOE) Test Execution Scientific Test and Analysis Techniques (STAT) Inform Decision-Makers The purpose of developmental testing is simple: to provide data to program leadership so that good decisions can be made as early as possible.

6 SE, DT&E, and DoDI Plan the Evaluation & Inform the Decisions Requirements Decision A Developmental RFP Decision B most important single decision point in the entire life cycle sets in motion everything that will follows C IOT&E Full Rate Production Decision Review MDD Materiel Solution Analysis Technology Maturation & Risk Reduction Engineering & Manufacturing Development Production and Deployment O&S PDR CDR LLP User Needs TEMP TEMP Draft TEMP DT&E TEMP Systems Engineering Management Plan Operational Evaluation Framework Concept of Operational Evaluation Satisfying the User Needs? Operations Did We Build the Right System? System Validation Operation & Maintenance Develop the Evaluation Strategy and identify the Critical Data needed to support key Technical, Programmatic, and Acquisition Decisions System Requirements System Evaluation Critical Data Needs System Verification & Deployment High-Level Subsystem Evaluation Developmental Design & Evaluation Framework Subsystem Satisfying Subsystem Technical Requirements and Informing Verification Development Requirements Process? Component Are We Building the System Evaluation Right? Detailed Design Unit Testing Software Coding Hardware Fabrication Developmental Evaluation Framework documents the information needed to inform key Technical, Programmatic and Acquisition decisions

7 DEF: Articulating the DT&E Strategy Articulate a logical evaluation strategy that informs decisions How acquisition, programmatic, technical and operational decisions will be informed by evaluation How system will be evaluated to generate the knowledge needed to inform decisions How test and M&S events will provide data for evaluation Decisions Define e Inform Evaluation What resources are required to execute test, conduct evaluation, and inform decisions Resources Schedule Define Data Test / M&S Define Exec ute DT&E story thread: decision evaluation test resources

8 Decision Support with an Evaluation Focus Capability questions Decision Support Questions (DSQ) System capabilities Technical measures DEO 1 DEO 2 DEO 3 TM 1 TM 2 *TM 3 KPP/KSA/CTP - related S t E i i d iti System Engineering decomposition: Evaluate system capability - Inform decisions

9 Decisions Defi ine In nform Evaluation Define Data Test / M&S Defin ne Execute Resources Schedule Developmental Evaluation Framework Developmental Evaluation Objectives Functional evaluation areas System capability categories Performance Performance Capability #1 Performance Capability #2 Interoperability Interoperability Capability #3 Interoperability Capability #4 Cybersecurity System Requirements and T&E Measures Technical Reqmts Document Reference Description 3.x.x.5 Technical Measure #1 DT#1 Decision #1 Decision #2 Decisions Supported Decision #3 Decision #4 DSQ #1 DSQ #2 DSQ #3 DSQ #4 DSQ #5 DSQ #6 DSQ #7 DSQ #8 Identify major decision points for which testing and evaluation phases, activity and events will provide decision supporting information. Cells contain description of data source to be used for evaluation information, for example: 1) Test event or phase (e.g. CDT1...) 2) M&S event or scenario 3) Description of data needed to support decision 4) Other logical data source description 3.x.x.6 Technical Measure #2 M&S#1 DT#3 3.x.x.7 Technical Measure #3 DT#3 3.x.x.8 Technical Measure #4 M&S#4 3.x.x.1 Technical Measure #1 DT#3 3.x.x.2 Technical Measure #2 IT#2 M&S#4 3.x.x.3 Technical Measure #3 3.x.x.4 Technical Measure #4 M&S#2 DT#4 M&S#2 DT#4 DT#4 DT#4 IT#1 IT#1 M&S#2 IT#2 IT#1 M&S#2 SW/System Assurance PPP 3.x.x SW Assurance Measure #1 SW Dev Assess SW Dev Asses SW Dev Assess RMF RMF Contol Measure #1 Cont Assess Cont Assess Cont Assess Cont Assess Vulnerability Assess Vul Assess Measure #1 Blue Team Blue Team Interop/Exploitable Vuln. Vul Assess Measure #2 Red Team Red Team Reliability 4xx1 4.x.x.1 Technical Measure #11 M-demo#1 Reliability Cap #1 4.x.x.2 Technical Measure #12 M-demo#1 IT#2 4.x.x.3 Technical Measure #13 M-demo#2 IT#2 Reliability Cap #2 4.x.x.4 Technical Measure #14 M-demo#2 IT#2 IT#1 DT#3 IT#5 IT#5 9

10 Then Plan the Test or Bringing it Back Together as IT With Evaluation Frameworks developed, informed integrated test planning can proceed Combine test resources (events, assets, ranges) Generate data to evaluate using DT or OT evaluation framework independent evaluation Inform DT or OT decision-makers different decisions How to design an analytically-rigorous ll i IT? At objective level, define common input factors/conditions, output measures of interest Develop input, process, output (IPO) diagram to illustrate IT design Apply DOE to generate common test cases 10

11 Decisions Defi ine In nform Evaluation Define Data Test / M&S Defin ne Execute Resources Schedule Developmental Evaluation Framework Developmental Evaluation Objectives Functional evaluation areas System capability categories Performance Performance Capability #1 Define Performance Capability #2 Interoperability Interoperability Capability #3 Interoperability Capability #4 Cybersecurity System Requirements and T&E Measures Technical Reqmts Document Reference Description 3.x.x.5 Technical Measure #1 DT#1 Decision #1 Decision #2 Decisions Supported Decision #3 Decision #4 DSQ #1 DSQ #2 DSQ #3 DSQ #4 DSQ #5 DSQ #6 DSQ #7 DSQ #8 Identify major decision points for which testing and evaluation phases, activity and events will provide decision supporting information. Cells contain description of data source to be used for evaluation information, for example: 1) Test event or phase (e.g. CDT1...) 2) M&S event or scenario 3) Description of data needed to support decision 4) Other logical data source description 3.x.x.6 Performance Technical Measure TPM#2 M&S#1 DT#3 3.x.x.7 Technical Measure #3 DT#3 3.x.x.8 Technical Measure #4 M&S#4 Interoperability 3.x.x.1 Technical TPM Measure #1 DT#3 3.x.x.2 Technical Measure #2 IT#2 M&S#4 3.x.x.3 Technical Measure #3 DOE-Based Test Design 3.x.x.4 Technical Measure #4 M&S#2 DT#4 M&S#2 DT#4 DT#4 DT#4 IT#1 IT#1 M&S#2 IT#2 IT#1 M&S#2 SW/System Assurance PPP 3.x.x SW Assurance Measure #1 SW Dev Assess SW Dev Asses SW Dev Assess RMF RMF Contol Measure #1 Cont Assess Cont Assess Cont Assess Cont Assess Vulnerability Assess Vul Assess Measure #1 Cybersecurity TPM Blue Team Blue Team Interop/Exploitable Vuln. Vul Assess Measure #2 Red Team Red Team Reliability 4xx1 4.x.x.1 Technical Measure #11 Reliability TPM Plan M-demo#1 Reliability Cap #1 4.x.x.2 Technical Measure #12 M-demo#1 IT#2 4.x.x.3 Technical Measure #13 M-demo#2 IT#2 Reliability Cap #2 4.x.x.4 Technical Measure #14 M-demo#2 IT#2 IT#1 DT#3 IT#5 IT#5 11

12 The Place for Design and Analysis of Experiments in T&E

13 Example - Space Based Infrared System (SBIRS) Space-based infrared sensors and ground-based d control and processing systems provide missile warning, missile defense, technical intelligence, and battlespace awareness 13

14 SBIRS DEF Space Segment Decisions Supported Ground Segment S2E2 Developmental Evaluation Objectives Functional evaluation areas System capability categories Performance HCS Reference System Requirements / Measures DEO/Capabilities SRD Reference Description Performance Probability of Warning* Probability of Collection* System Mission Performance Coverage* Minimum Threat* System Mission ; Focused Area* Performance Data Collection ; Data Availability* Mission Simultaneity Reporting Attack Magnitude False Track Rate Line of Sight Radiometric Accuracy Mission Functionality Data Collection Closely Spaced Object (Min Spatial Resolution) System Survivability/Endurability ; Report Data and Contents 3.1.4; ; Report Time* Missino Reporting Management False Report Rate See to Ground Focus Area* Data Fidelity Interoperability Data Archiving Data Fusion ; Track Telemetry and Control Mission interface ; compliance Commanding Autonomous Ephemeris Failover/Activiation Timelines Functionality Net ready Anomaly Resolution ; Set up Cybersecurity Text Message Handling MGS Backwards Compatability 3.4 Environmental Characteristics 3.6 Space Segment Survivability/Endurability System/SW ; 3.6 Ground Segment 3.6 Assurance Design and Construction Surveillance Msn Data Processing Priorities Mission Management Compliance (RMF) Managing System Resources Collaborative OPIR Functionality Interoperability Vulnerability Assessment Mission Interface USNDS Compliance External Comms Net Ready Net Ready KPP* Reliability Suitability Logistics Cybersecurity System/SW Assurance Cybersecurity Measure #1 Risk Management Cybersecurity Measure #2 Vulnerability Assessment Cybersecurity Measure #3 Reliability Suitability Logistics Dependability Reliability Maintainability EMC Human Factors Supportability Ground Segment Loading Availability ILS support 3.5.5; Personnel & training SV Flight Readiness SV performance and stability sufficient for launch? SV ops acceptance Tuning and on-orbit performance sufficient to accept into constellation? Ground (Blk 20) readiness (PEO Certification) Blk 20 performance satisfy HCS requirements? Govt DT TRR Does S2E2 operate through environmental requirements? Decisions & DSQs Inc 1 (DSP) PEO Certification Mono/DSP performance satisfy S2E2 SRD requirements? Inc 2 (GEO) PEO Certification GEO performance satisfy S2E2 SRD requirements? Space Segment Data Sources (Test, M&S events) SV Flight Readiness SV performance and stability sufficient to launch SV Ops Acceptance Tuning & on-orbit performance sufficient to accept into constellation? Ground Segment Ground (Blk 20) readiness (PEO Certification Blk 20 performance satisfy HCS requirements? S2E2 Govt DT TRR Does S2E2 operate thru environmental reqmts? Inc 1 (DSP) PEO Certification Mono/DSP performance satisfy S2E2 SRD reqmt? Inc 2 (GEO) PEO Certification GEO performance satisfy S2E2 SRD reqmt? Requirements/Technical Measures High Component Spec (HCS), S2E2 SRD 14

15 SBIRS DOE Working Group Enterprise-wide SME Involvement STAT The Ballistic Missile Threat Background Infrared Radiation Operations Modeling & Simulation i Systems Engineering Testers "An approximate answer to the right problem is worth a good deal more than an exact answer to an approximate problem." -- John Tukey

16 SBIRS COI 1 Factor Space Factor Name COI 1 Factors (Version April 2014) OLGASim NG LM Factor Factor Factor Type Levels Factor Rank Rank Subtype Management Level Descriptors Missile Type 1 x 1 1 Categorical Nominal 5 Vary ICBM, SLBM, IRBM, MRBM, SRBM Attack Magnitude 1 Categorical Nominal 3 Vary Small, Medium, Large Threat Categorical Nominal 4 Vary None, A, B, C (Demo A, B, C) Source Missile Intensity 2 x Numeric Continuous 2? Min, Max Burn Duration 2 x Categorical Nominal 3? Short, Intermittent, Long 2 Missile il Acceleration x Numeric Continuous 2? Min, Max Launch Origin Lattitude x 4 2 Numeric Continuous 129 Vary Min, Max Launch Origin Longitude x Numeric Continuous 129 Vary Min, Max Aim Point Lattitude x 4 Numeric Continuous 1108 Vary Min, Max Aim Point Longitude x Numeric Continuous 1108 Vary Min, Max True Launch Azimuth x 2 Numeric Continuous n/a Log -180, +180 Local Zenith Angle x 4 Numeric Continuous n/a Log 0, +180 Solar Season 3 Categorical Nominal 2 Vary Eclipsed, Non-Eclipsed Time of Day 1 3 Categorical Nominal 2 Vary Day, Night Cloud Cover 3 Categorical Nominal n/a Log Cirrus, Cirrostratus, Cirrocumulus, None Atmospheric Transmission 3 HEO Coverage Categorical Ordinal n/a Log 0, 1, 2 Sensor Type 2 1 Categorical Nominal n/a Log GEO Scanner, GEO Starer, HEO, DSP, Combination, Other Data Sensor with Sufficient Angle Categorical Nominal n/a Log 0 thru N (N = Classified) Launch Notice 4 Categorical Nominal 3 Vary None, Short, Advanced Operator Experience Categorical Nominal n/a Log Begginer, Intermediate, Advanced Number of Strategic Events Numeric Continuous n/a Log Classified Concurrent Strategic Events Numeric Continuous n/a Log Classified Release Mode 4 Categorical Nominal n/a Log Operator, Auto Release Communication Link Categorical Nominal n/a Log Given Factor Type Threat Trajectory Environmental Constellation Operational 16

17 Tech perf EFs Defines IT Data Needs DT EF OT EF Ops perf questions Crit Dev Issue (DSQ) questions CitO Crit Ops Issue (COI) Technical capabilities Operational capabilities DEO 1 DEO 2 DEO 3 Technical measures TM 1 TM 2 *TM 3 KPP/KSA related OT Obj 1 OT Obj 2 OT Obj 3 Operational measures MOE 1 MOE 2 *MOS 3 Some are KPPs Potential common data for IT 17

18 IT Design Example IPO Diagram Controlled Factors Covariates Responses Simulated missile type Launch point lat/long Aim point lat/long Missile Warning (Operational Capability) Launch point/time accuracy Impact point/time accuracy OT Measures Constellation geometry IR background Report Content (Performance Requirements) State vector accuracy Report time (3) accuracy DT Measures Elevation angle Noise Uncontrollable Factors Ref: Beers, S. M., Brown, C. D., Cortes, L. A. (2014). The E before the efficient & rigorous T : From Developmental Evaluation Framework to Scientific Test and Analysis Techniques implementation. ITEA Journal 2014; 35:

19 Summary & Way Ahead E : DEF focuses system evaluation to inform decisions DSQ (decision) i DEO (capability) TM (measure) T : DOE & STAT adds rigor to the T&E strategy and feeds the DEF Plan (measures, factors, test design) Execute (test t points) Data (statistical analysis) Inform Decisions e Defin Inform Evaluation Define DOE Test Design Execute STAT Data Analysis Defi ine Data.all others must bring data. Resources Schedule 19

Developmental Evaluation Framework & STAT-based Integrated Test Design

Developmental Evaluation Framework & STAT-based Integrated Test Design Dr. Suzanne Beers NDIA T&E Conference 22-23 July 2014 Approved for Public Release; Distribution Unlimited. Case Number 14-1801. 2014