Lockheed Martin Benefits Continue Under CMMI CMMI Technology Conference 2004 November 17, 2004 Joan Weszka Lockheed Martin Corporate Engineering & Technology Systems & Software Resource Center. CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.
Context Agenda Background on Lockheed Martin s (LM) CMMI transition approach Benefits in the Software CMM and CMMI eras LM Systems Integration (Owego, NY) LM Maritime Systems & Sensors Tactical Systems (Eagan, MN) LM Maritime Systems & Sensors Radar Systems (Syracuse, NY) LM Maritime Systems & Sensors Undersea Systems (Manassas, VA) Joan Weszka - 2
Context Lockheed Martin organizations that tracked quantitative process improvement benefits during their SW-CMM high maturity journey have now transitioned to CMMI Previously reported data showed these benefits continued with CMMI implementation This trend continues in 2004 However, benefits derived are not attributable only to CMMI Many initiatives are underway concurrently with CMMI deployment (and were underway when the SW- CMM was in use) CMM is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. Joan Weszka - 3
Key Tenets of Lockheed Martin s CMMI Transition Address CMMI in the context of your organization s business requirements Lockheed Martin s Integrated Engineering Process (LM-IEP) standard includes CMMI, in addition to other standards and requirements (e.g., ISO/IEC 15288, ISO 9001:2000) Use SCAMPI SM A for benchmarking Adopt an incremental appraisal approach for process improvement and SCAMPI SM readiness Lockheed Martin Continuous Appraisal Method (CAM) has been successfully deployed with CMMI SM SCAMPI is a service mark of Carnegie Mellon University. Joan Weszka - 4
LM-IEP Context Diagram ANSI/EIA-632 ISO 9001:2000 CMMI V1.1 Industry Standard Gov t Standard Domain Specific Standards IEP Integrated Engineering Process LM Lockheed Martin CPS Corporate Policy Statement LM CPSs ISO/IEC-12207 LM Hardware Process Standard LM-IEP Standard Organizational Standard Process(es es) ISO/IEC-15288 IEEE 1220 AS9100 Industry Standard Gov t Standard Project Specific Standards Project Defined Process Joan Weszka - 5
Continuous Appraisal Method (CAM) Design Goals Minimize appraisal preparation and reduce cost Integrate process improvement with process appraisal activities Facilitate appraisal scheduling and minimize disruption for participants Provide an appraisal environment conducive to process improvement Promote institutionalization CAM was originally developed for use with EIA/IS 731 and is being used with CMMI Joan Weszka - 6
Overview of Incremental Appraisal Using CAM Planning Appraise OSP Fix Weaknesses Re-appraise OSP Appraisal Appraise PA 1 Fix Weaknesses Re-appraise Appraise PA 1 Fix Weaknesses Re-appraise Project Appraisals OSP = Organizational Standard Process PA = Process Area Appraise PA 1 Fix Weaknesses Maintenance Review Institutionalization focus with minimal project disruption... Re-appraise Joan Weszka - 7
Feedback on CAM usage 14 CAMs have been completed or are underway at Lockheed Martin operating units using CMMI 6 prior CAMs have been completed using EIA/IS 731 Experience with CAM has been positive: More focus on process improvement Facilitated by the incremental approach Less invasive to programs Less stressful to the organization More value-add, in-depth findings More cost effective Effective method to prepare for SCAMPI SM SM SCAMPI is a service mark of Carnegie Mellon University. Joan Weszka - 8
Lockheed Martin Systems Integration - Owego Process Credentials 2003 CMMI Level 5 / AS 9100A 2003 2002 Q-100 Award 2002 2001 ISO 9001:2000 / SEI CMMI Pilot Site 2000 Lean / Six Sigma 1998 LM21 Best Practices 1997 SW CMM Level 5 2001 2000 1998 1997 1995 RIT / USA TODAY Quality Cup Finalist 1994 1996 ISO 9001:1994 / SW CMM Level 3 New York State Excelsior Award 1993 IBM Baldrige Gold Award 1996 1995 1994 1993 1990 TQM 1990 Joan Weszka - 9
Lockheed Martin Systems Integration Owego - Software Productivity 140.0% 120.0% 100.0% 80.0% 60.0% 40.0% 20.0% Improvements Since 1993: 12.7% Average per Year SW CMM Level 3 Capable Software Productivity (All Software including Reuse) ISD Processes SW CMM Level 3 SW CMM Level 5 CMMI Level 3 Capable Contributors: CMMI Level 5 Increased Reuse (Domain Specific) Process Maturity and Compliance Process Consistency Increased use of High Order Language / 4th Generation / Object Oriented Use of development and test tools 0.0% 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 Joan Weszka - 10
Lockheed Martin Systems Integration Owego - Quality Software Defects per Million Delivered Source Lines of Code % Reduction STRs / MSLOC 0% -20% -40% -60% -80% Integrated Processes SW CMM Level 3 SW CMM Level 5 Improvements Since 92 80.5% Overall CMMI Level 3 Capable CMMI Level 5 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 Note: Started counting * STR / MSLOC defects = Software differently Trouble in 2004. Report/Million RedefinedSource from Lines Latent of Code to Open at Delivery. CMMI and CMM are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. Joan Weszka - 11
LM Maritime Systems & Sensors Tactical Systems Process Improvement Credentials Oct. 1999 Attained SW-CMM level 4 Dec. 2000 Attained Systems Engineering Capability Model (EIA 731) level 3 Jan. 2001 Began focus on integrated process improvement Began transition to CMMI June 2002 May 2003 CAM Appraisal OSP: Target profile 5 for CMMI -SE/SW/IPPD/SS Projects: Target profile 3 for CMMI -SE/SW August 2003 SCAMPI SM Appraisal Achieved maturity level 3 using CMMI -SE/SW CMM is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. Joan Weszka - 12
LM Maritime Systems & Sensors Tactical Systems 2004 Initiatives Implement CMMI -SE/SW maturity level 4 Streamlining of references and guidance documents which supplement the OSP Conduct LM-IEP gap analysis Risk management methodology best practice Populate Process Asset Library (PAL) Self audit process compliance Mechanical engineering guidebook Airworthiness manual Work product templates Cost Estimation Relationships Guide Joan Weszka - 13
LM Maritime Systems & Sensors Tactical Systems Software Productivity 3.50 Source Statements per Hour 3.00 2.50 2.00 1.50 1.00 0.50 0.00 Level 3 Level 4 Level 3 CMM CMM CMMI 1979 1984 1989 1994 1999 2004 Year Joan Weszka - 14
LM Maritime Systems & Sensors - Syracuse Process Improvement Credentials SW-CMM level 5 (CBA IPI SM ) in Dec. 1999 Systems Engineering Capability Model (EIA/IS 731) level 3 (CAM) in Dec. 1999 Focus on integrated process improvement including hardware began in 2000 Transition to CMMI began in 2000 CMMI -SE/SW/IPPD target profile 4 (CAM) in Nov. 2002 CMMI -SE/SW maturity level 4 (SCAMPI SM ) in May 2004 SW-CMM and CMMI are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. SM SCAMPI is a service mark of Carnegie Mellon University. Joan Weszka - 15
LM Maritime Systems & Sensors - Syracuse 2004 Functional Excellence Initiatives LM21 Business Excellence Design for Six Sigma (DFSS) Technical Assurance Integrated Product Development Digital Dashboard CMMI -SE/SW Maturity Level 4 Continuous Process Improvement Process Development Quantitative management Joan Weszka - 16
LM Maritime Systems & Sensors - Syracuse Software Productivity Software Engineering Productivity History (Completed, Current/In-Process, Projected-Goal, BQM) 35.0 30.0 25.0 SLOCs / Staff Day 20.0 15.0 10.0 5.0 0.0 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 SW SW SW SECM CMMI CMMI CMM CMM CMM L3 CAM SCAMPI L3 L4 L5 L4 L4 Completed Log Best-Fit Joan Weszka - 17
LM Maritime Systems & Sensors Undersea Systems Process Credentials Systems & Software Engineering* Software CMM Level 4 (CBA IPI) - June 1995 Software CMM Level 5 (CBA IPI ) - February 1999 CMMI & EIA-731 target profile 3 (CAM) - October 2001 CMMI -SE/SW/IPPD/SS target profile 5 (CAM) - October 2002 Quality Management ISO 9001:1994 - September 1995 AS9000 - November 1997 Defense Contract Management Agency (DCMA) ISO 9001 Qualified - December 1997 ISO 9001:2000 - December 2000 AS9100A December 2002 Certificate No. Certificate No. Certificate No. 10061 7092 9227 Certificate No. Certificate No. EN10002 E00015 *Assessed programs compriseover over80% of ofthe Undersea Systems development programs, and andall allparts partsof ofthe developmentcycle. Joan Weszka - 18
LM Maritime Systems & Sensors Undersea Systems Process Chronology 1970s Top-down Structured Programming Design & Code Inspections 1980-2 Functional Decomposition SW Engineering Workshop Advanced Design Workshop 1983 SW Management Workshop 1984 Ada Workshop 1985 Requirements Inspections 1986 FSC Practices & Measurements 1988 SW Technology Steering Group Organizational Operating Procedures 1990 SW Engineering Process Group Formed First SW-CMM Assessment (Level 3) Formal Estimation Procedures 1991 Market Driven Quality Reuse Focus 1992 Defect Prevention Process 1993 Integrated Teams Standard Development Environment Integrated Process Group 1994 Automated Metrics Process Coordination Group (PCG) 1995-6 Integrated Process Library ISO 9001 Registration Software CMM Level 4 1997-9 ISO 14001 Registration, AS9000 & DCMA ISO 9001 qualification Software CMM Level 5 2000-> ISO 9001: 2000 Certification EIA-731 Level 3 AS9100A Certification CMMI -SE/SW/IPPD/SS Target Profile 5 (CAM) Joan Weszka - 19
LM Maritime Systems & Sensors Undersea Systems Software Product Quality Software Product Quality (Defects Per Million Delivered Source Statements) 3,000 2,500 CMMI L5, 10/02 Defects Per MDSS 2,000 1,500 1,000 CMM L3, 9/90 CMM L4,6/95 CMM L5,2/99 CMMI L3, 10/01 500 0 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 Program Completion Year Note: CMM levels were achieved via CBA IPI. CMMI levels indicate the result of CAM assessments using CMMI -SE/SW/IPPD/SS with a lead appraiser outside of Undersea Systems. Joan Weszka - 20
LM Maritime Systems & Sensors Undersea Systems Software Quality S ig ma De fe cts /MS 1 690,000.0 2 308,537.0 3 66,807.0 4 6,210.0 5 233.0 6 3.4 MS = Million Source Statements Product Quality Level is in Five Sigma Range Def/Million Parts 8 0 0,0 0 0 7 0 0,0 0 0 6 0 0,0 0 0 5 0 0,0 0 0 4 0 0,0 0 0 3 0 0,0 0 0 Our quality rate is 20 times better than the average industry rates. 2 0 0,0 0 0 5 Sigma 1 0 0,0 0 0 0 1.0 0 2.0 0 3.0 0 4.0 0 5.0 0 6.0 0 S ig m a N u m b e r Joan Weszka - 21
Joan Weszka - 22 Reference: A Correlational Study of the CMM and Software Development Performance Lawlis, Flowe & Thordahl, CROSSTALK, September 1995 SCATTER PLOTS BETWEEN 1 AND 3 RATINGS ARE INDUSTRY SAMPLES. LOCKHEED MARTIN DATA HAS BEEN ADDED IN RED. LM Maritime Systems & Sensors Undersea Systems Cost & Schedule Estimates vs. Actuals at Maturity Level 5 DATA FROM 8 PROGRAMS IN 2Q 2003 Cost Performance Index (CPI) & Schedule Performance Index (SPI) 0 1 2 3 1 2 3 4,5 0 1 2 3 1 2 3 4,5 = = = = = = = = = = = C P I S P I CMM Rating CMM Rating
LM Maritime Systems & Sensors Undersea Systems Process Maturity Benefits Continuous process improvement is rooted in Manassas culture...and the benefits are so ingrained in our approach, we tend to take them for granted: Consistency More likely to get the same results each time within a small range THIS IS A VERY REAL MEASURE OF QUALITY IN TODAY S MARKETPLACE Predictability Much easier to estimate work Better able to assess impact of change Manageability Able to measure and track progress Better able to change and improve Better able to control costs and meet schedule IT ALL ADDS UP TO IMPROVED QUALITY IMPROVED PROFITABILITY Joan Weszka - 23
Example of Defect Discovery Profile for Another LM Organization Example Defect Discovery Goals & Actuals To Date Normalized Defects Per KSLOC 4.50 4.00 3.50 3.00 2.50 2.00 1.50 1.00 0.50 0.00 Low er Range Values Goal, Expected Value Upper Range Values Actuals 90%Confidence Range HLD LLD C CT/UT/ST RT/ESI/SWIT I&T Latent Verification Stage Joan Weszka - 24
Summary The trend previously reported continues: Benefits derived during SW-CMM implementation continued to be realized as CMMI maturity evolves Allocating benefits to their sources is difficult when implementing multiple models/standards and initiatives SW-CMM and CMMI are viewed as significant (but not sole) contributors to process improvement benefits to date Joan Weszka - 25
Contact Information Joan Weszka Lockheed Martin Corporate Engineering & Technology Systems & Software Resource Center (301) 240-7013 / Fax 240-7009 joan.weszka@lmco.com Joan Weszka - 26
Acronyms ARC Appraisal Requirements for CMMI CAM - Continuous Appraisal Method CMM - Capability Maturity Model CMMI - Capability Maturity Model Integration ESLOC or ESS Equivalent SLOC/SS; a normalized value derived from new development, plus SLOC/SS that are modified, retained, ported, etc. IPPD - Integrated Product and Process Development LM - Lockheed Martin LM-IEP - Lockheed Martin Integrated Engineering Process OSP Organizational Standard Process PA - Process Area SLOC Source Line of Code SS Source Statement (sometimes called a Logical SLOC ) Joan Weszka - 27