The Business Case for Systems Engineering: Comparison of Defense-Domain and Non- Defense Projects

Transcription

1 The Business Case for Systems Engineering: Comparison of Defense-Domain and Non- Defense Projects Presenter: Joseph P. Elm The Software Engineering Institute (SEI) a DoD Research FFRDC

2 Copyright 2014 Carnegie Mellon University This material is based upon work funded and supported by the Department of Defense under Contract No. FA C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN AS-IS BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. This material has been approved for public release and unlimited distribution except as restricted below. This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu. Carnegie Mellon is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. DM

3 Challenges in DoD Acquisition GAO T - Actions Needed to Overcome Longstanding Challenges with Weapon Systems Acquisition and Service Contract Management costs increased 26% and development costs increased by from first estimates programs failed to deliver capabilities when promised often forcing warfighters to [maintain] legacy systems current programs experienced, on average, a 21-month delay in delivering initial capabilities to the warfighter Although DoD is the largest acquirer in the world, acquisition troubles remain MDAP RDT&E cost growth (mean) 84% 2011 MDAP Procurement cost growth (mean) 28% Effectiveness ( ) 89% Suitability ( ) 72% Nunn-McCurdy breach rate from % 1. Performance of the Defense Acquisition System 2013 Annual Report Table 2-3, page 34) 3

4 Root Cause of Poor Program Performance Inadequate Systems Engineering! MDAP Cost Growth: PARCA Root Cause Analysis 1 Finding from Performance of the Defense Acquisition System 2013 Annual Report Dominant root cause of MDAP Cost Growth Finding from GAO T 10 of 18 (56%) 5 of 18 (28%) Dominant Poor management Systems engineering Contractual incentives Risk management Situational Awareness Baseline cost and schedule estimates Framing assumptions managers rely heavily on assumptions about system requirements, technology, and design maturity, which are consistently too optimistic. These gaps are largely the result of a lack of a disciplined systems engineering analysis prior to beginning system development 4 of 18 (22%) Change in procurement quantity Infrequent 1 of 18 Immature technology, excessive manufacturing, or integration risk 2 of 18 Unrealistic expectations 1 of 18 Unanticipated design, engineering, manufacturing or technology issues None Funding inadequacy 1. Performance of the Defense Acquisition System 2013 Annual Report Table 2-3, page 34) 4

5 Why Do We Fail to Utilize Good SE Practices? It s difficult to justify the costs of SE in terms that project managers and corporate managers can relate to. The costs of SE are evident Cost of resources Schedule time The benefits are less obvious and less tangible Cost avoidance (e.g., reduction of rework from interface mismatches) Risk avoidance (e.g., early risk identification and mitigation) Improved efficiency (e.g., clearer organizational boundaries and interfaces) Better products (e.g., better understanding and satisfaction of stakeholder needs)` We need to quantify the effectiveness and value of SE by examining its effect on project? 5

6 The 2012 SE Effectiveness Study Purpose Strengthen the business case for SE by relating project to the use of SE practices. Method Contact development projects using the resources of NDIA, AESS, and INCOSE. Survey projects to assess their SE activities Project Degree of challenge Process responses to identify statistical relationships between parameters. Survey Tenets All data is submitted anonymously and handled confidentially by the SEI. Only aggregated non-attributable data is released. 6

7 The Bottom Line: SE = Performance 15% 24% 47% 52% 29% 56% 24% 2 Across ALL projects, 1/3 are at each level For Lower SEC projects, only 15% deliver higher For Middle SEC projects, 24% deliver higher Gamma = 0.49 p-value < For Higher SEC projects, 57% deliver higher Gamma = 0.49 represents a VERY STRONG relationship 7

8 For Challenging Projects SE is even MORE important Perf vs. SEC_Total (Low PC) Perf vs. SEC_Total (High PC) % 23% 45% 32% Lower SEC (n=22) 58% 19% Middle SEC (n=26) 52% 36% 12% Higher SEC (n=25) % 23% 69% Lower SEC (n=26) 26% 35% 39% Middle SEC (n=23) 62% 12% 27% Higher SEC (n=26) Gamma = 0.34 p-value = Gamma = 0.62 p-value = A STRONG relationship between Total SE and Project Performance for CHALLENGE projects A VERY STRONG relationship between Total SE and Project Performance for CHALLENGE projects 8

9 Study Participants Participant Solicitation Contacted key members of major defense contractors to promote study participation Contacted the memberships of NDIA SE Division, IEEE AESS, and INCOSE Collected 148 valid responses Ind. Mfg & Svc: defense Which of these best describes your industry or service? Ind. Mfg & Svc: Electronic Ind. Mfg and Svc: Other Transportation Energy 9 Communications 0 1 Consumer Goods & Svc Health Care 10 Other Please enter the country in which most of the design and development engineering will be/was performed. 130 USA UK South Africa Australia Canada India The Netherlan Sweden Finland 9

10 and Performance SYSTEMS ENGINEERING DEPLOYMENT Risk Management Product Integration Integ Product Teams Total SEC Early SE Project Planning Req'ts Dev't & Mg't PROJECT PERFORMANCE 3.70 PerfS Validation Verification Monitor & Control Trade Studies Configuration Mg't Product Architecture DEF Non-DEF Perf PerfT DEF PerfC Non- DEF 10

11 Total SE vs. Project Performance Project Performance vs. Total SE (defense) % 35% 54% 42% 61% 31% 23% 15% Lower SEC (n=23) Middle SEC (n=26) Higher SEC (n=26) Total Systems Engineering Capability (SEC) Gamma = 0.57 Project Performance vs. Total SE (non-defense) A Very Strong relationship between applied SE and Project Performance for both Defense and non-defense Projects % 25% 25% 55% 38% 67% 36% 38% 9% Lower SEC (n=12) Middle SEC (n=8) Higher SEC (n=11) Total Systems Engineering Capability (SEC) Gamma =

12 Architecture vs. Project Performance Perf vs. SEC_ARCH (defense) % 37% 47% 48% 35% 17% Lower SEC (n=19) Middle SEC (n=33) Higher SEC (n=23) Gamma = 0.38 A Strong relationship between Architecture activities and Project Performance for Defense Projects A Very Strong relationship for nondefense projects Perf vs. SEC_ARCH (non-defense) % 29% 57% 45% 36% 18% Lower SEC (n=14) Middle SEC (n=6) Higher SEC (n=11) Gamma =

13 Requirements Dev t & Mg t vs. Performance Perf vs. SEC_REQ (defense) % 19% 23% 55% 54% 27% 56% 26% 19% Lower SEC (n=22) Middle SEC (n=26) Higher SEC (n=27) Gamma = 0.46 A Very Strong relationship between Requirements activities and Project Performance for both Defense and non-defense Projects Perf vs. SEC_REQ (non-defense) % 58% 2 67% Lower SEC (n=12) Middle SEC (n=10) Higher SEC (n=9) Gamma =

14 Risk Management vs. Project Performance Perf vs. SEC_RSKM (defense) % 25% 28% 44% 46% 27% 29% 23% Lower SEC (n=25) Middle SEC (n=28) Higher SEC (n=22) Gamma = 0.28 A Moderate relationship between Risk Management activities and Project Performance for Defense Projects A Very Strong relationship for nondefense projects Perf vs. SEC_RSKM (non-defense) % 36% 43% 13% 13% 75% 56% 44% Lower SEC (n=14) Middle SEC (n=8) Higher SEC (n=9) Gamma =

15 Trade Studies vs. Project Performance Perf vs. SEC_TRD (defense) % 52% 36% 36% 28% 48% 39% 13% Lower SEC (n=27) Middle SEC (n=25) Higher SEC (n=23) Gamma = 0.45 A Very Strong relationship between Trade Study activities and Project Performance for Defense Projects A Strong relationship for nondefense projects Perf vs. SEC_TRD (non-defense) % 29% 57% 27% 44% 22% Lower SEC (n=7) Middle SEC (n=15) Higher SEC (n=9) Gamma =

16 Summary of Relationships -1 Performance vs. SE Capability Total SEC Early SE Project Planning Req'ts Dev't & Mg't Verification Product Architecture Configuration Mg't Trade Studies Monitor & Control Validation Product Integration Risk Management Integ Product Teams Project Challenge Experience Moderate Weak Moderate Strong Very Strong DEF NonD 16

17 Summary of Relationships -2 Next Steps: Investigate the differences between SE deployment / effectiveness in defense and non-defense domains to find transplantable best practices 17

18 Questions for Further Study On non-defense projects, why are SE activities in Requirements, Architecture, Risk Management, and Verification more effective than those on defense-related projects? On defense projects, why are SE activities in Trade Studies, IPTs, and Project Monitoring and Control more effective than those on non-defense projects? Why is the relationship between Project Challenge and Project Performance stronger for non-defense projects? Why is the relationship between Prior Experience and Project Performance stronger for non-defense projects? 18

19 Next Steps Download the 2012 SE Effectiveness reports from the SEI website The Business Case for Systems Engineering Study: Results of the Systems Engineering Effectiveness Survey The Business Case for Systems Engineering Study: Detailed Response Data The Business Case for Systems Engineering Study: Assessing Project Performance from Sparse Data The Business Case for Systems Engineering: Comparison of Defense Domain and Non-Defense Projects Search for ways to apply the findings within your own work and your own organization. Contact the SEI with questions or to obtain assistance. 19

20 SEI Your Resource for Software and Systems Engineering For more information, contact Software Engineering Institute 4500 Fifth Avenue Pittsburgh, PA OR Joseph P. Elm

21 BACK UP

22 References Elm, J.; Goldenson, D.; El Emam, K.; Donatelli, N.; Neisa, A. A Survey of Systems Engineering Effectiveness Initial Results. Carnegie Mellon University; Pittsburgh, PA (available at Elm, J.; Goldenson, D. The Business Case for Systems Engineering Study: Results of the Systems Engineering Effectiveness Survey. Carnegie Mellon University; Pittsburgh, PA 2012 (available at Elm, J.; Goldenson, D. The Business Case for Systems Engineering Study: Detailed Response Data. Carnegie Mellon University; Pittsburgh, PA 2012 (available at Elm, J. The Business Case for Systems Engineering Study: Assessing Project Performance from Sparse Data. Carnegie Mellon University; Pittsburgh, PA 2012 (available at Elm, J.; Goldenson, D. The Business Case for Systems Engineering: Comparison of Defense Domain and Non-Defense Projects. Carnegie Mellon University; Pittsburgh, PA 2014 (available at 22

23 IPT Utilization vs. Project Performance Perf vs. SEC_IPT (defense) % 38% 46% 39% 45% 35% 26% 25% Lower SEC (n=24) Middle SEC (n=31) Higher SEC (n=20) Gamma = 0.32 A Strong relationship between IPT Utilization and Project Performance for Defense Projects A Moderate relationship for nondefense projects Perf vs. SEC_IPT (non-defense) % 36% 55% 25% 25% Lower SEC (n=11) Middle SEC (n=8) Higher SEC (n=12) Gamma =

24 Project Planning vs. Project Performance Perf vs. SEC_PP (defense) % 32% 59% 32% 43% 25% 56% 28% 16% Lower SEC (n=22) Middle SEC (n=28) Higher SEC (n=25) Gamma = 0.57 A Very Strong relationship between Project Planning activities and Project Performance for both Defense and non-defense Projects Perf vs. SEC_PP (non-defense) % 23% 69% 43% 45% 29% 29% 45% 9% Lower SEC (n=13) Middle SEC (n=7) Higher SEC (n=11) Gamma =

25 Verification vs. Project Performance Perf vs. SEC_VER (defense) % 55% 24% 41% 34% 58% 31% 12% Lower SEC (n=20) Middle SEC (n=29) Higher SEC (n=26) Gamma = 0.55 A Very Strong relationship between Verification activities and Project Performance for both Defense and non-defense Projects Perf vs. SEC_VER (non-defense) % 73% 2 2 Lower SEC (n=11) Middle SEC (n=10) Higher SEC (n=10) Gamma =

26 Validation vs. Project Performance Perf vs. SEC_VAL (defense) % 32% 35% 35% 53% 32% 17% Lower SEC (n=17) Middle SEC (n=34) Higher SEC (n=24) Perf vs. SEC_VAL (non-defense) Gamma = 0.45 A Very Strong relationship between Validation activities and Project Performance for both Defense and non-defense Projects % % 2 43% 2 Lower SEC (n=5) Middle SEC (n=21) Higher SEC (n=5) Gamma =

27 Product Integration vs. Project Performance Perf vs. SEC_PI (defense) % 25% 58% 47% 47% 7% Lower SEC (n=12) Middle SEC (n=48) Higher SEC (n=15) Gamma = 0.45 A Very Strong relationship between Product Integration activities and Project Performance for both Defense and non-defense Projects Perf vs. SEC_PI (non-defense) % 43% 38% 29% 38% 29% 25% Lower SEC (n=9) Middle SEC (n=14) Higher SEC (n=8) Gamma =

28 Configuration Mg t vs. Project Performance Perf vs. SEC_CM (defense) % 52% 45% 42% 31% 42% 24% 16% Lower SEC (n=27) Middle SEC (n=29) Higher SEC (n=19) Gamma = 0.42 A Very Strong relationship between Configuration Management activities and Project Performance for both Defense and non-defense Projects Perf vs. SEC_CM (non-defense) % 58% 36% 27% 36% 38% 13% Lower SEC (n=12) Middle SEC (n=11) Higher SEC (n=8) Gamma =

29 Monitoring & Control vs. Project Performance Perf vs. SEC_PMC (defense) % 36% 37% 54% 31% 15% Lower SEC (n=22) Middle SEC (n=27) Higher SEC (n=26) Gamma = 0.48 A Very Strong relationship between Project Monitoring and Control activities and Project Performance for Defense Projects A Strong relationship for nondefense projects Perf vs. SEC_PMC (non-defense) % 22% 67% 2 Lower SEC (n=9) Middle SEC (n=12) Higher SEC (n=10) Gamma =

30 Prior Experience vs. Project Performance Perf vs. EXP (defense) % 42% 35% 29% 35% 55% 18% 27% Lower EXP (n=24) Middle EXP (n=40) Higher EXP (n=11) Experience Gamma = 0.1 A Weak relationship between Prior Experience and Project Performance for Defense Projects A Strong relationship for nondefense projects Perf vs. EXP (non-defense) % 17% 67% 42% 25% 43% 43% 14% Lower EXP (n=12) Middle EXP (n=12) Higher EXP (n=7) Experience Gamma =

31 Project Challenge vs. Project Performance Perf vs. PC (defense) % 25% 31% 31% 31% 38% 38% 31% Lower PC (n=20) Middle PC (n=29) Higher PC (n=26) Project Challenge Gamma = A Weak Negative relationship between Project Challenge and Project Performance for Defense Projects A Moderate Negative relationship for non-defense projects Perf vs. PC (Non-defense) % 67% 17% 63% 38% 18% 18% 64% Lower PC (n=12) Middle PC (n=8) Higher PC (n=11) Gamma = Porjects delivering 31

32 Early SE vs. Project Performance Perf vs. Early_SE (defense) % 29% 58% 32% 36% 32% 57% 39% 4% Lower Early SE (n=24) Middle Early SE (n=28)higher Early SE (n=23) Gamma = 0.61 A Very Strong relationship between Early SE activities and Project Performance for both Defense and non-defense Projects % 25% 67% Lower Early SE (n=12) Middle Early SE (n=9) Early SE Project Planning Requirements Development Trade Studies Product Architecture Perf vs. Early_SE (non-defense) 1 Higher Early SE (n=10) Gamma =