CMMI Level 5: Return on Investment for Raytheon N TX. Donna Freed Network Centric Systems, McKinney, TX

Transcription

1 CMMI Level 5: Return on Investment for Raytheon N TX Donna Freed Network Centric Systems, McKinney, TX

2 Achieving CMMI Level 5 We did it! How Did We Do It? Achieve Engineering Goals. This presentation describes the benefits of achieving CMM Level 4 in 21, and then to achieve CMMI Level 5 in /1/24 Page 2

3 We Did IT! Raytheon North Texas is the first site in Raytheon and fifth company in the world to achieve CMMI Level 5. Measurable results are achieved before achieving Level 5. This Presentation shows the actual ROI of going to each level, as well as our ROI projection. 12/1/24 Page 3

4 How Did We Use CMMI to Achieve? Why is Raytheon North Texas pro-active about achieving CMMI Level 5? Because we want to achieve the performance excellence goals required by our business. We are focused on achieving performance excellence and recognition as the preferred supplier for new business. 12/1/24 Page 4

5 Envision Improvement Integrated Product Teams: Cross-functional resources to implement our processes Capability Maturity Model Integration: The yardstick for judging the maturity of our processes IPTs IPTs Integrated Product Development System: Where we define our product development processes Raytheon Six Sigma: How we improve our processes Programs Integrate R62, IPDS and CMMI into their Pland IPTs Product teams use common tools and processes in an environment of continuous improvement guided by industry Best Practices 12/1/24 Page 5

6 How Did We Achieve Performance Goals? How did we use CMMI to achieve our performance excellence goals? We picked performance goals that were important to us. The metrics data we collect characterizes the organizational performance in terms of our organizational goals and identifies opportunities of improvement. 12/1/24 Page 6

7 SWEC SWIP Objectives Meet Commitments (to Customer) Intent: Meet the cost and schedule objectives of the programs we support. Quantification: CPI and SPI SW Price Intent: Price software engineering products competitively Quantification: $ / DLOC Deliver Quality Intent: Deliver quality software engineering products Quantification: In-phase Defects and Defect Density We have been executing statistical process control on the overall process using these measures for years. 12/1/24 Page 7

8 Organization Process Analysis frequency Process Capability for CPI LSL =.975, Nominal = 1., USL = CPI Cp =.9 Cpk = -.3 Cpk (upper) =.2 Cpk (lower) = -. Cr = Cpm =.9 K = -.61 Process Tooling People Too program Multiple data Subjectivity Procedure focused, need sources Tools & (differing points product line & process to of view) org view autom ate defect capture W hat counts H ow to coun t (definition of defects) defects in reused code Clear guidelines for phases of orig Peer review process Criteria variation S everity Type & rea son Phas es Pjt to pjt differences in counting R oles C onducting Meeting & follow-up Corrective ac tion Varying types Needs to include: Inconsistent Training Type & reason codes not useful Not a priority on a program Peer review training Purpose not clear No Org Training JIT Doors Continuus Defect Logger Tools not user friendly Defect Doublecounts?? Continuus learning curve People need to know the concept of operations N ew people do n ot know how to classify defects Effective meeting training needed for peer reviews People don't know how to use the defect data Source No common repository for shared code Round trip eng. (i.e. integrated tooling) Domain Variation Types Program s Mix of program s Old/New Reason code Defect Existence Phase of origin Tim e to do prework Schedule Constraints Customer care abouts Always start at 1% Using correct people in reviews SW Experts Systems Limited Domain Experts People concerned about defect counts being used against them O verloaded A ttrition Defect C ontainm ent Metric has Excessive Variation: s > 28% > 48 >42 and <=48 >36 and <=42 >3 and <=36 >24 and <=3 >18 and <=24 > 12 and <=18 > 6 and <= 12 <= 6 SW Program Duration (M onths) Mean = 23 Median = 18 Min = 5 Max =12 STD EV = 22 n = $/DLOC Regression Model ELOC/DLOC Use R6 Tools for Metrics Analysis Dollars per DLOC

9 Improvement Results Demonstrated the linkage between R6 and CMMI Levels 4 & 5. Characterization included over 3 applications of R6 tools such as ANOVA, cause and effect, regression analysis, histograms, Cpk, hypothesis testing, logical process mapping, and others. Identified five projects to reduce variation in organizational performance and support the CMMI Level 5 timeline. Enabled CMMI Level 5 certification. Improvement of Business Performance was recognized by Assessment Team as global strength in the CMMI Level 5 Assessment. Contributed ROI of 3:1 through significant cost avoidance realized by organization improvements 12/1/24 Page 9

10 Operational Results Achieving CMMI Level 5 Certification for Raytheon image and competitive advantage is one thing, but look at the operational results. Meeting Commitments all improved concurrent with SEI CMMI Level 5 certification Across the organization, we improved: CPI by 5 percentage points, and reduced variation by 34%. SPI by 8 percentage points, and reduced variation by 5% Defect Density by 44 percentage points, and reduced variation by 31% 12/1/24 Page 1

11 Cost Performance Index Process Capability for CPI frequency January 24 CPI Process Capability for CPI frequency February 21 2 We no longer have CPI special cause variation on the low end! CPI 12/1/24 Page 11

12 Cost Performance Index CMM Level 4 June 21 SWEC CPI Trend CMMI Level 5 September 23 Feb 1 Mar 1 Apr 1 May 1 Jul 1 Sep 1 Dec 1 1Q2 2Q2 3Q2 4Q2 1Q3 2Q3 3Q3 4Q3 Upper DROV Nominal value Lower DROV Mean Performance Improved CPI by 5 percentage points, and reduced variation by 34%. 12/1/24 Page 12

13 Schedule Performance Index Process Capability for SPI frequency January 24 SPI Process Capability for SPI frequency February 21 3 SPI We are still moving in the right direction! 12/1/24 Page 13

14 Schedule Performance Index CMM Level 4 June 21 SWEC SPI Trend CMMI Level 5 September 23 Feb 1 Mar 1 Apr 1 May 1 Jul 1 Sep 1 Dec 1 1Q2 2Q2 3Q2 4Q2 1Q3 2Q3 3Q3 4Q3 Upper DROV Nominal value Lower DROV Mean Performance Improved SPI by 8 percentage points, and reduced variation by 5% 12/1/24 Page 14

15 Defect Density Process Capability for Defect Density frequency January 24 Defect Density Process Capability for Defect Density frequency February 21 1 Defect Density Continuing improvement in mean and variation. 12/1/24 Page 15

16 Defect Density CMM Level 4 June 21 SWEC Defect Density Trend CMMI Level 5 September 23 Feb 1 Mar 1 Apr 1 May 1 Jul 1 Sep 1 Dec 1 1Q2 2Q2 3Q2 4Q2 1Q3 2Q3 3Q3 4Q3 Upper DROV Nominal value Lower DROV Mean Performance Improved Defect Density by 44 percentage points, and reduced variation by 31% 12/1/24 Page 16

17 Characterize CPI Analysis Plot of Regression Model CPI Percentage Organization Process Adherence Regression Analysis included a sample of various process characteristics. Projects that follow the standard process tend to have a better and more predictable CPI performance. Process adherence is not a guarantee of CPI success. It improves the probability of CPI success. 12/1/24 Page 17

18 Characterize SPI Analysis Observed SPI Predicted SPI based on Process Adherence vs. Observed SPI Multiple Regression Analsysis Prediction of SPI Multiple Regression Analysis included Level 2, Level 3, and Level 4 process characteristics. Projects that follow the standard process tend to have a better and more predictable SPI performance. Process adherence is not a guarantee of SPI success. It improves the probability of SPI success. Organizational process adherence is the only identified factor affecting SPI. 12/1/24 Page 18

19 Results Our improvements were recognized as organizational strengths in the appraisal. This accomplishment leads the way for Raytheon to distinguish ourselves from the competition and achieve customer satisfaction through superior program execution. There is no higher illustration of customer focus than this level of excellence. Colin Schottlaender, Raytheon NCS President These improvements contributed to ROI of 3:1 12/1/24 Page 19