Enhanced Systems Engineering - Starting Programs Right NDIA 11 th Annual Systems Engineering Conference October 23, 2008 Ceasar D. Sharper Systems and Software Engineering/Enterprise Development Office of the Deputy Under Secretary of Defense (Acquisition & Technology)
Outline Enhanced Systems Engineering (SE) SE Context: Background / Framework Early SE: right activities at the right time - Materiel Solution Analysis Phase - Technology Development Phase Emphasis on SE.. the right time in the right way - Competitive Prototyping - SE Design Consideration - Reliability, Availability, and Maintainability - Preliminary Design Review (PDR) Implementing the the right right activities at at the the right right time time in in the the right rightway 2
Background Program Roles & Activities Stakeholder Management Planning Cost Management Schedule Management Configuration Management Contract Management Concept Selection Architecture Development Requirements Baseline Technical Risk Management Interface Control Integration Verification Validation Project Manager Primary Primary Primary Primary Primary Primary Shared Shared Systems Engineer Shared Primary Primary Primary Primary Primary Primary Shared 3
Technology Opportunities & Resources User Needs Proposed DoDI 5000.02 Changes Framework for Enhanced SE Materiel Development Decision precedes entry into any phase of the acquisition system Entrance criteria met before entering phase Evolutionary Acquisition or Single Step to Full Capability Strategic Guidance Joint Concepts FAA FNA JCD FSA Materiel ICD MDD Solution Analysis MS A TechDev MS B CDD Engineering & Manufacturing Development & Demonstration MS C CPD Full Rate Prod DR Production & Deployment O&S OSD/JCS COCOM FCB AoA Incremental Development Preliminary Design Review Precedes MS B Right activities at the right time MS A MS B MS B MS C FRP DR MDD Materiel Solution Analysis Technology Development SRR PDR Engineering & Manufacturing Development & Demonstration CPD PD O&S CDD CDD 4
Processes Technical Technical Processes Material Solution Analysis MS A Acquisition Acquisition Phases Phases Technology Development MS B Engineering Manufacturing Development and Demonstration Production and Deployment MS C Operations and Stakeholder Requirements Definition Requirements Analysis Architecture Design Implementation Right Right activities at at the the right right time time Verification Transition Validation Technical Technical Reviews Reviews ITR ASR SRR SFR PDR CDR TRR SVR/ FCA/ PRR OTRR PCA ISR Technical Technical Baseline Baseline Preferred System Concept System Spec System Functional Baseline Allocated Baseline Initial Product Baseline Product Baseline Early Systems Engineering ITR - Initial Technical Review ASR- Alternative Systems Review SRR - System Requirements Review SFR System Functional Review PDR Preliminary Design Review CDR Critical Design Review SVR System Verification Review TRR- Test Readiness Review FCA- Functional Configuration Audit PRR Product Readiness Review OTRR Operational Test Readiness Review PCA Physical Configuration Audit ISR In-Service Review
Materiel Solution Analysis Phase MDD Material Solution Analysis MS A JCIDS ICD Initial user assessment of capability needs Government Program Office Systems Engineering Engineering Analysis of Preferred Systems Solution(s) PSC ITR Study Efforts ASR Technical Planning for MS A SEP Other Government Program Office Activities Areas Depending on SE input TES TD RFP(s) TD Plan TDS Typically executed by PMO SE Staff AoA Guidance AoA Plan AoA Report Typically executed by Industry Delivered Product Conduct AoA Alternative Systems Review Grey Areas depending on PMO SE input Leads to Informs Key SE input 6
SE Activities During MSA Phase Systems Engineering Processes/Documents/Plans Key Technical Processes Systems Engineering Plan (SEP) Test and Evaluation Strategy (TES) Analysis of Alternatives (AOA) Input to the Technology Development Strategy Input to the Cost Estimate Assessments Program Review (PSR) Technical Reviews Initial Technical Review (ITR) Alternative System Review (ASR) SE COP (https://acc.dau.mil/techrevchklst). 7
Technology Development Phase MS A Technology Development MS B JCIDS User assessment of capability needs CDD Execute Tech Maturation Initial TRA Requirements to Preliminary Design** Develop Feasible System Design (FD) FD Draft System Level Spec Sys Spec PDR(s) PDR Report TRA CTE Risk Reduction * CTE Prototyping Update SEP Update SEP SRR Design Prototyping SFR PDR SE Prep for MS B Final System Spec SEP ISP Other Program Activities Actions Depending on SE input Typically executed by PMO SE Staff Typically executed by Industry Delivered Product Draft RFP for Initial Sys Dev Initial AS TEMP CARD/ICE CCE APB Update TEMP CARD/ICE CCE APB Update RFP* Update AS TEMP CARD/ICE CCE APB Final AS TEMP CARD/ICE CCE APB Mandated Preliminary Design Review Technical Review Technical Assessment Grey Areas depending on PMO SE input Leads to Informs Key SE input * Prototyping for CTE and for design may be independent efforts ** May vary with contracting strategy (e.g., multiple designs) 8
SE Activities During TD Phase Systems Engineering Processes/Documents/Plans Key Technical Processes Competitive Prototyping Technology Maturation Test and Evaluation Master Plan (TEMP) Cost Analysis Requirements Description (CARD) Input to the Acquisition Program Baseline (APB) Assessments Technology Readiness Assessment (TRA) Program Review (PSR) Technical Reviews Systems Requirements Review (SRR) Systems Functional Review (SFR) Preliminary Design Review (PDR) 9
Prototyping and Competition in the right way Evolutionary acquisition requires... Technology development preceding initiation of an increment shall continue until the required level of maturity is achieved, prototypes of the system or key system elements are produced, and a preliminary design is completed.... The TDS and associated funding shall provide for two or more competing teams producing prototypes of the system and/or key system elements prior to, or through, Milestone B. The prototypes shall be representative platforms reflecting the maturity of technologies and integrated system performance consistent with expected capability. 10
Competitive Prototyping Done in the Right Way Need to know earlier on what will make the program successful and prototype that (i.e. challenges) Decide what is important cost, integration, technology, etc and determine how to measure / assess success Cost in prototyping should be a factor but the not dominant decision point Get domain experts to assist in determining what needs to be prototyped Do proof of concept but also to fill in the other holes Achieved at any level system or key system elements (sub-system, assembly, or component) Prototype the critical path items first Need to spend money smartly up front get smart at low burn rate 11
Enhanced Systems Engineering SE Design Consideration - RAM Reliability, Availability, & Maintainability (RAM) Defense Science Board Report on DT&E (dtd May 08) recommended to improve RAM DoD Working Group formed to implement recommendations Reliability, Availability, and Maintainability Policy (dtd 21 Jul 08); Directs Components to set policy actions to ensure: Collaboration in the establishment of RAM requirements Development contracts and acquisition plans evaluate RAM during system design Maturation of RAM throughout the acquisition life cycle Use of contract incentives to achieve RAM goals (http://www.acq.osd.mil/sse/dte/docs/usd-atlmemeo-ram-policy-21jul08.pdf) 12
Implementing RAM-C right activities at the right time 13
Implementing RAM-C in the right way SE Design consideration in the right way Template for Reliability Contract Language Sections C, L, and M Guidance on Performance Incentives for Reliability GEIA-STD-0009, Reliability Program Standard for Systems Design, Development, and Manufacturing RAM Planning Template by each Technical Review Evaluation Criteria (Reliability Program Detailed Scorecard) to assess a program Early T&E Involvement in RFP Development DoD Reliability, Availability, Maintainability and Cost Rationale Report Manual, October XX, 2008 (http://www.acq.osd.mil/sse/dte/spec-studies.html) Having performance is is important, but but not not as as important in in most cases, as as having reliability 14
Life Cycle Logistics Flow (RAM) Evaluate Technology s Ability to Implement ability Objectives & Sustainment Strategy Refine Sustainment Objectives and Requirements Set Sustainment (RAM) Metrics & Requirements Design-in Sustainment Features & Establish Product Package Requirements - CONOPS - CONOPS - Analysis of Alternatives - Analysis of Alternatives - Capability Description Doc - Capability Description Doc - Capability Production Doc - Capability Production Doc - Production/Quality Metrics - Production/Quality Metrics -Operational Use Data -Operational Use Data Implement Product Strategy and Package Logistics And Sustainment Thread Define Initial ability Objectives & Sustainment Strategy Demonstrate Feasibility & Mature Technology Design Product Package Develop Product Package & Demonstrate Critical Capabilities Monitor Performance and Adjust Product User Needs A Materiel Concept Refinement Solution Analysis Concept Decision Materiel Development Decision Technology Development Technology Development TRA B System Development & Production & Demonstration Deployment Engineering & System Design System Demonstration LRIP/IOT&E/Full Rate Production & Deployment & FRP Decision Review Manufacturing Development & Demonstration C Deployment IOC FOC Operations & Sustainment Operations & Disposal Reviews ITR ASR SRR IBR SFR PDR CDR TRR FRR SVR/FCA/ OTRR PCA ISR Technical Baseline Preferred System Concept System Specification System Functional Baseline Allocated Baseline Product Baseline RAM included in Systems Engineering Tech Reviews Product Baseline Contract Action
Enhanced Role of PDR MS A MS B MS C FRP DR SRR MDD Materiel Solution Analysis Technology Development SRR SFR PDR CDR P-CDRA EMDD CPD PD O&S Interpret User Needs, Refine System Performance Specs & Environmental Constraints Trades Develop System Functional Specs & Verification Plan To Evolve System Functional Baseline Revised Program Information Technical Plans IMS/IMP Risk Assessment Detailed Requirements CDD Acquisition Program Baseline Acquisition Strategy SEP / Technical Planning Functional Baseline SFR TEMP / Test Plans Evolve Functional Performance Specs into CI Functional (Design to) Specs and CI Verification Plan Cost Estimate (ICE//EA CARD) PPP / CPI Sys Allocated Baseline Live Fire T&E Waiver request (if appropriate) Risk Assessment Sys Threat Assessment Affordability Assessment Cost & Manpower Est. PDR Allocated Baseline Information collected from PDR enhances plans & estimates to enable knowledge-based MS-B decision and certifications Information Plan CDD / Requirements Certifications (e.g., CCA) PESHE Post PDR Report TRA (TRL Level 6 / Off-Ramp) 16
PDR in the right way Certification and Accreditation activities scoped and identified Configuration Management Plan and procedures scoped and implemented Integrated Master Schedule showing Critical Path through Critical Design Review Software Development Plan scoped and documented at the Configuration Item level FMECA scheduled to support System Hazard Analysis Modeling and Simulation role in testing and life cycle planning scoped Representative mission profiles finalized 17
PDR in the right way (Continued) Engineering data requirements needed from testing identified Data element identification procedures established IDE procedures established Test Verification Matrix covering subsystem allocations Physical properties (i.e., weight, power, cooling, etc.) allocated to subsystems Human Systems Integration design standards flowed to subsystems R&M diagnostics addressed in design allocations Interface Control Documents between subsystems completed 18
PDR Report to MDA DRAFT PDR Report Guidance to require the following: A comprehensive list of the systems engineering products that make up the Allocated Baseline, per the PDR review, A list of the participants in the review. including the independent (of the program) chair, applicable technical authorities, independent subject matter experts, membership of the Technical Review Board, and other key stakeholders, A summary of the Action Items and their closure status/plan A resulting risk assessment using a PDR risk assessment checklist and readiness to commit to full detail design, A recommendation from the PDR as to the approval of the program s system Allocated Baseline to support detail design. Proposed Source: DAG para 4.3.2.4.2.3 19
Enhanced SE Provides key information for the MS B Decision Enhanced SE contributes to key MS B prerequisites Acquisition Strategy (including core logistics analysis/source of repair; cooperative opportunity; etc.) Independent Cost Estimate Cost Analysis Requirements Description (CARD) Manpower estimate Acquisition Program Baseline Analysis of Alternatives System Threat Assessment Technology Readiness Assessment (TRA) Affordability Assessment Selected Acquisition Report (SAR) SEP, TEMP, Program Protection Plan, and PESHE Clinger-Cohan Act compliance 20
Enhanced SE Summary Enhanced Systems Engineering is the lynchpin to start programs right! Early SE in support of MDD, MS A, and B SE activities in support of Technical Reviews and essential program planning efforts Implementing SE.in the right way Competitive Prototyping Reliability, Availability, Maintainability Cost implementation Implementing the the right right activities at at the the right right time time in in the the right rightway Defense Acquisition Guidebook (DAG) ( http://akss.dau.mil/dag/) The Systems Engineering Community of Practice (https://acc.dau.mil/communitybrowser.aspx?id=17608); 21
Backup 22
RAM Improvement Efforts Some RAM Pitfalls to avoid when executing a sound systems engineering process include: Inadequate planning for reliability and maintainability Failure to identify mission context or intended use profile when stating RAM requirements Failure to design-in reliability early Reliance on predictions instead of design analysis Inadequate lover level testing Lack of proper planning, managing, and executing reliability growth activities, and Lack of reliability incentives 23
SE Provides a Technical Foundation for Acquisition Business Decisions Engineering MDD Agreement to pursue a material solution Uncertainty Preferred System Analysis AoA Material Solution Analysis MS A Selection of a preferred solution Preferred System Concept Technology Maturation And Prototyping System Level Specs PDR Technology Development Preliminary Design MS B Formal Program Start CDR Completed Design National Research Council Pre-Milestone A and Early- Phase Systems Engineering Jan 2008 Systems Engineering is most effective when it initiated early to start a program right! 24
Phases Tech Reviews Work Efforts Material Developer MDD RAM-C Activities FY 08 FY 09 FY 10 FY 11 FY 12 FY 13 FY 14 FY 15 FY 16 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 Material Solution Technology Engineering and Manufacturing Production & Deployment Operations & Analysis A Development BDevelopment & DemonstrationC OTRR TRA TRA IBR OTRR OTRR IOC FOC SFR CDR ITR ASR SRR PDR TRR SVR/PRR PCA ISR Material Development Decision Post-CDR Assessment Pre-FRP Developmental Testing Technical Feasibility Tests EUT Prod Prove-out Tests LUT Write TDS Write APB/ASR Engineering Development Tests SW Tests Write SEP Update SEP Update TEMP/SEP Write TES Write TEMP ID Key R/M Cost Drivers (CTPs/TPMs) FD/SC feedback Test Report Feedback Reliability Allocations Revised RAM input Feasibility analysis LRIP Decision Review PQT / LFT Log Demo IOT LRIP Deliveries FRP Decision Review Production Testing Post-Production DT PVT / CPT Acceptance Insp. Surv./Stockpile Reliability Reconditioning Tests Combat Developers Reports AoA Guidance CONOPS RAM goals ICD Initial AoA Develop COICs (with assumptions) Trades Write OMS/MP Write FD/SC Quantify RAM DRAFT ID Maintenance Concepts Update AoA (update assumptions) CDD EUT Assessment Report Trades Validate RAM Update AoA Initial DT Report CPD Includes a RAM-C Rationale Report LUT Report Includes Executive Summary of the RAM-C Rationale Report Interim DT Report IOT&E/ BLRIP/ Final DT Reports Interop/NR Certification LF Report to Congress DT OT Software Build/LRIP Delivery Tech Reviews Program Reviews Technology Readiness Assessment
Pre-Milestone B Sustainment Requirement Process Stakeholders A Technology Development B Combat Developer Finalize OMS/MP Develop FD/SC -Quantify RAM requirements (KPP) -Assess validity of assumptions Update to AoA Draft CDD CDD RAM-C Executive Summary Program Manager (To include: Systems Engr. Logistics ) Input Trades Input SRR -Develop Product Strategy/Plan -Develop System Performance Specification -Develop Life Cycle Cost Analysis verified with analysis, tests (Tech Demos), or comparison -Perform CAIV Analysis -Develop CTP/TPMs -Conduct feasibility analysis -Manufacturing risks identified/assessed -Manage prototype development State of the Art Analysis Input Develop TEMP Update Trades Input to AoA SEP SFR Develop APB Input PDR Develop ASR APB ASR Joint Staff J8/J4 J8 Provide APB Input (JROC Interest Program) Approve CDD Service Staff Update AoA Updated AoA Includes a RAM-C Rationale Report Validate CDD OSD -Oversight/Review Systems Engineering Activities -Oversight/Review Program Documentation Approve TEMP Provide APB Input OD/PA&E Independent Assessment for Milestone Decision T&E Organizations Provide FD/SC Input TECH DEMOS Provide TEMP Input Evaluate Prototypes