How Do We Get On The Road To Maturity? Debra J. Perry Harris Corporation NDIA Conference - 1
What do we want? From this To this But how? NDIA Conference - 2
Where Do We Start? NDIA Conference - 3
Government Communications Systems Division: What We Do Aviation electronics Communications and information networks Intelligence, surveillance, and reconnaissance Space and ground satellite communications systems Operations and support services We innovate, integrate, and manage technology. NDIA Conference - 4
Roadmap to Getting There 1 of 3 ROI Need Convince management of payoff Convince other stakeholders management is convinced Convince other stakeholders process improvement helps them NDIA Conference - 5
Roadmap to Getting There 2 of 3 ROI Need Convince management of payoff Convince other stakeholders management is convinced Convince other stakeholders process improvement helps them Convince Management Share process improvement vision Provide external ROI SEI data Other companies data Provide internal ROI Harris examples Other possibilities NDIA Conference - 6
Roadmap to Getting There 3 of 3 ROI Need Convince management of payoff Convince other stakeholders management is convinced Convince other stakeholders process improvement helps them Convince Management Share process improvement vision Provide external ROI SEI data Other companies data Provide internal ROI Harris examples Other possibilities Convince Other Stakeholders Show management support Educate and train Show value for them NDIA Conference - 7
ROI for Management ROI Return on Investment Convince Management of Payoff External Research Read and Research Attend SEI Courses Hire consultants Internal Research Develop a plan Develop ROI Educate management Start or improve your process group NDIA Conference - 8
The Process Management Premise The Quality of a System is Highly Influenced by the Quality of the Process Used to Acquire, Develop and Maintain It This Premise Implies a Focus on the Processes as well as on Products: This is a long-established premise in manufacturing Belief in this premise is visible worldwide in quality movements in manufacturing and service industries (e.g., ISO standards) This premise is also applicable to development (Introduction to CMMI Version 1.2) NDIA Conference - 9
Ad Hoc Processes Processes are Not Well Defined and Improvised by Practitioners and their Management Process Descriptions are Not Rigorously Followed or Enforced Performance is Highly Dependent on Current Practitioners Understanding of the Current Status of a Project is Limited Immature Processes Result in Fighting Fires: There is no time to improve instead practitioners are constantly reacting Firefighters get burned Embers might rekindle later (Introduction to CMMI Version 1.2) NDIA Conference - 10
Improved Processes Process Descriptions are Consistent with the Way Work is Actually Done They are Defined, Documented and Continuously Improved Processes are Supported Visibly by Management and Others They are Well Controlled Process Fidelity is Evaluated and Enforced There is Constructive Use of Product and Process Measurement Technology is Introduced in a Disciplined Manner (Introduction to CMMI Version 1.2) NDIA Conference - 11
Benefits for Improving Processes Processes Enable you to Understand What is Going on People Develop their Potential More Fully and are More Effective Within the Organization By Defining, Measuring and Controlling the Process, Improvements are More Successful and Sustained The Likelihood that Appropriate Technology, Techniques and Tools are Introduced Successfully Increases More Benefit Information is Available in the August 2006 SEI Technical Report, Performance Results of CMMI- Base Process Improvement (CMU/SEI-2006-TR-004) at: http://www.sei.cmu.edu/publications/documents/06.report s/06tr004.html (Introduction to CMMI Version 1.2) NDIA Conference - 12
ROI At Different Levels Process Improvement Start Up Reaching Next Level of Maturity Out of Phase Defect Removal Reach Higher Levels of Maturity Optimization and Maintenance ROI Should be More Specific as you Move Up in Maturity ROI Difficulty Costs Profiles of Level 5 CMMI Organizations Little has been done to study the return on investment (ROI) of high maturity organizations that have reached Level 5 NDIA Conference - 13
ROI for Other Stakeholders 1 of 2 Convince Other Stakeholders Management is Convinced Show management is investing in process improvement Show management is enforcing compliance Management should fund and provide oversight for process improvement efforts and review and reward process compliance. NDIA Conference - 14
ROI for Other Stakeholders 2 of 2 Convince Other Stakeholders Management is Convinced Show management is investing in process improvement Show management is enforcing compliance Convince Other Stakeholders Process Improvement Helps Them Relieves chaos Reduces defects and rework Facilitates improvements Saves time and money Toyota makes process improvement a way of life. NDIA Conference - 15
Convince Management Share Process Improvement Vision Provide External ROI SEI data Other companies data Provide Internal ROI Harris examples Other possibilities NDIA Conference - 16
Share The Vision Find a Strong Leader with Clout and Credibility Develop a Vision What Will Motivate? Fewer problem programs Faster, cheaper, better Need CMMI to compete Develop a -Plan Determine Estimated ROI Present the Plan to Management Be Determined! Get Management Commitment NDIA Conference - 17
Example Process Improvement Vision Enhance the Current Processes to: Improve efficiency and value for all stakeholders Reduce cost and rework Provide predictable program execution Increase competitive advantage NDIA Conference - 18
Process Improvement Steps Establish Process Group, if Possible Document Current Processes Institutionalize Processes Determine Process Improvement Goals Short term goals (start with PP, PMC, MA) Long term goals (add PAs, plan for SCAMPI) Estimate ROI Prioritize Process Improvement Goals Develop Process Improvement Plan Determine process improvement measurements Begin measurement collection efforts NDIA Conference - 19
Performance Results of CMMI -Based Process Improvement 2006 Improvements High Low Median Number Data Points Cost 87% 3% 34% 29 Schedule 95% 2% 50% 22 Productivity 329% 11% 61% 20 Quality 132% 2% 48% 34 Customer Satisfaction 55% -4% 14% 7 Return on Investment 27.7:1 1.7:1 4.0:1 22 NDIA Conference - 20
LM Customer Satisfaction: Award Fees Award fees increased by 55% compared to an earlier SW-CMM ML2 baseline. 100% Per Cent Potential Additional Award Fee SW CMM L2: 1993-1995 SW CMM L3: 1996-1997 SW CMM L4: 1998-1999 SW CMM L5: 2000-2001 CMMI L5: 2002 Additional Award Fee Achieved = 55% of the Potential Additional Award Fee Award Fees vs. CMM Level SW CMM L2 SW CMM L3 SW CMM L4 SW CMM L5 CMMI L5 Customer Satisfaction Continues to Improve NDIA Conference - 21
NG ROI Example 1 of 2 Defect Prevention Using PSP and CAR at CMMI ML5 Defect Density (Defects/KLOC) 7 6 5 4 3 2 1 0 6.6 DP 1 CAR 6.1 DP 2 CAR 3.5 DP 3 CAR 3.9 1 2 3 4 5 Build 2.1 Improvements in: Quality Integrating PSP sm and CMMI Level 5. Gabriel Hoffman, Northrop Grumman IT. May 1, 2003 NDIA Conference - 22
NG ROI Example 2 of 2 Appraised at CMMI ML 5 in December 2002 Results met 25+ milestones in a row earned a rating of Exceptional in every applicable category on a formal Contractor Performance Evaluation Survey Hours Invested: 124 in Defect Prevention (CAR) Hours saved: 1650 hours (15 hours per defect) ROI: 13:1 Improvements in: Schedule / cycle time Customer satisfaction Quality Cost of quality / ROI Integrating PSP sm and CMMI Level 5. Gabriel Hoffman, Northrop Grumman IT. May 1, 2003 NDIA Conference - 23
External ROI Ericsson Ltd. PROCESS IMPROVEMENT Support Expenditures Training costs Execution of Inspections Total Costs Savings from Inspections Total Cost Savings $207,000 $69,000 $358,000 $634,000 $2,524,000 $1,890,000 NDIA Conference - 24
External ROI Others page 1 of 2 NDIA Conference - 25
External ROI Others page 2 of 2 Industry Published SCAMPI Results Federal Contractor Level 4 Level 5 Delta 6/06 BAE 3 0 Boeing 11 3 Computer Sciences Corporation 2 1 General Dynamics 5 3 Lockheed Martin 3 4 0/2 Northrop Grumman (including Mel.) 36 15 Raytheon 2 5 0/0 SAIC 7 2 Sikorsky Aircraft 1 0 NDIA Conference - 26
Harris ROI Example Program Collected Metrics Program Analyzed Metrics Program Made Process Improvement Changes Process Changes Improved Metrics NDIA Conference - 27
Level 4 Quantitative Management What Is QM? Indicates a problem Control chart Voice of the Process Predictable & quantitatively understood Statistics & quantitative techniques to manage processes & results Address Special Causes Level 5 Original zone of quality control Capability Control Chart Chronic waste Quality Improvement Quality improvement New zone of quality control New zone of quality control Less chronic waste Voice of the Customer Continual & measurable process improvement to meet business objectives Incremental & innovative process improvements Address Common Causes NDIA Conference - 28
Process Improvements L4/L5 Example Defect Reduction GCSD Modified Engineering Objective: Monitor Measurement key functional Objectives processes Monitor for key predictability functional processes and for measurable predictability improvement and measurable (e.g., 10% improvement productivity, (e.g., 20% productivity, quality) quality) 90% AF 97.8% AF 100% AF 100% AF 30 Rejections County Rejections (Count by (Count Month) by Month) First time rejections 25 20 15 10 5 0 Oct-03 Oct-03 Oct-03 Oct-03 Nov-03 Nov-03 Dec-03 Nov-03 Nov-03 Dec-03 Dec-03 Dec-03 Jan-04 Feb-04 Jan-04 Jan-04 Feb-04 Feb-04 Jan-04 Feb-04 Mar-04 Mar-04 Mar-04 Mar-04 Apr-04 Apr-04 Apr-04 Apr-04 May-04 May-04 May-04 May-04 Jun-04 Jun-04 Jun-04 Jun-04 Jul-04 Jul-04 Jul-04 Jul-04 Aug-04 Aug-04 Aug-04 Aug-04 Sep-04 Sep-04 Sep-04 Sep-04 Oct-04 Oct-04 Oct-04 Oct-04 Nov-04 Nov-04 Nov-04 Nov-04 Dec-04 Dec-04 Dec-04 Dec-04 Jan-05 Jan-05 Jan-05 Jan-05 Feb-05 Feb-05 Feb-05 Feb-05 Mar-05 Mar-05 Mar-05 Mar-05 Apr-05 Apr-05 Apr-05 Apr-05 May-05 May-05 May-05 May-05 Jun-05 Jun-05 Jun-05 Jun-05 Jul-05 Jul-05 Jul-05 Jul-05 Aug-05 Aug-05 Aug-05 Aug-05 Sep-05 Sep-05 Sep-05 Sep-05 Oct-05 Oct-05 Oct-05 Oct-05 Nov-05 Nov-05 Nov-05 Nov-05 Dec-05 Dec-05 Dec-05 Dec-05 Jan-06 Jan-06 Jan-06 Jan-06 Number of Rejections Number of UCL Rejections Mean LCL Current Data Changes made: Team personnel changes - Intuitively suggests that process has improved over time Stakeholder participation - Predictable? formalized - Measurable improvement? Peer Review process Stable performance (within control limits*) Predictable average of 9.1 rejections per month (capability) deployed more formally Return to Green * not enough data points to classify as statistical control but can use trial limits Re-calculated limits due to process changes Stable performance (within tighter control limits*) Predictable average of 2.5 rejections per month (capability) shows 73% measurable improvement NDIA Conference - 29
Process Improvements L4/L5 Example CPI 1.80 1.60 1.40 TEAM & PROCESS IMPROVEMENTS MADE CPI CPI Average Upper Control Limit Lower Control Limit Upper Control Limit 1.51 Upper Control Limit 1.52 CPI 1.20 1.00 0.80 0.60 Average 1.02 Lower Control Limit 0.54 Average 1.12 Lower Control Limit 0.72 0.40 0.20 0.00 200310 200312 200402 200404 200406 200408 200410 200412 200502 200504 200506 200508 200510 200512 200602 200604 90% AF 98% AF 100% AF 100% AF 100% AF After process improvements implemented: Average cost performance improved 10% Lower end of predictable performance range improved 33% BENEFITS: Higher productivity, cost stability, higher award fees NDIA Conference - 30
Internal ROI Ideas Rework ROI Estimate rework cost Estimate defect reduction Estimate cost avoidance UPS ROI Estimate cost for downtime Determine cost of UPS Estimate savings Automation ROI Estimate manual effort Estimate automated effort Estimate savings Other Suggestions? NDIA Conference - 31
Convince Other Stakeholders Show Management Support Educate and Train Motivate to change Communicate vision Teach Tools and Techniques Reward compliance Show Value for Them NDIA Conference - 32
Show Management Support What You Measure You Will Improve. author unknown Harris Process Compliance Monitor Monitors all required processes Appropriate tailoring allowed Artifacts of compliance entered into tool Online product-centric evidence collection, objective auditing Online real-time project and organizational monitoring of process compliance Required compliance NDIA Conference - 33
Process Compliance Color Scores ASSESSMENT STATUS COLORS NY NA NS Not Yet Not Applicable Not Scored To be appraised at a later date (i.e., the process has not yet been executed by the program and cannot be appraised) Outside the scope of the project (e.g., Code and Unit Test Process is not applicable to a production-type program) Pending an appraisal FI Fully Implemented Direct artifacts are present and appropriate (Note 2) No weaknesses noted (Note 1) PROCESS COMPLIANCE COLORS LI PI NI Largely Implemented Partially Implemented Not Implemented Direct artifacts are present and appropriate (Note 2) One or more weaknesses noted (Note 1) Direct artifacts are missing in the initial scoring audit or direct artifacts are present but inadequate (Note 3) One or more weaknesses noted (Note 1) Direct artifacts are missing for more than 30 days from the initial scoring audit Note 1: Note 2: Note 3: A weakness ("gap") is considered if it is an impact to or risk of implementation of the process statement An appropriate artifact is the IPM Expected Artifact or equivalent that demonstrates implementation of the process statement An inadequate artifact does not demonstrate implementation of the process statement NDIA Conference - 34
Process Compliance Represents overall process compliance score for program Based on lowest color score harsh, but in keeping with CMMI standards Depicts scoring distribution over all process items More insight on overall project score A B Depicts score for each process executed or being executed by this program 3 columns identify types of processes In PCM, point+click on underlined acronym drills down to scoring details for process C NDIA Conference - 35
Educate User Community Practical Software and Systems Measurement SEI Training Courses on Process Improvement Six Sigma Training Provide Technical Expertise Train Them to: Follow the processes Collect consistent measures Analyze data Suggest improvements Train for Appraisals NDIA Conference - 36
Show Value for Stakeholders Process Improvement Proposal: Business Intelligence Implementation Invest in Tool, Save in Usage Current Estimates for Creating Metrics and Reports Monthly ~$1.5 million per Year Current Estimate for Incorporating BI ~$2M Time Saved Each Month Can be Used Elsewhere Other benefits Consistent, timely, accurate data Flexibility in reporting Easier new analysis NDIA Conference - 37
Summary Without a Vision, There Can Be no Improvements State, Share and Spread the Vision Of Process Improvement Management Acts Based on the Bottom Line Show and Tell how process improvement can improve their business (ROI) Other Stakeholders Tend to Follow Management Convince Other Stakeholders that management is convinced and is watching Teams Want Improvement Show and Tell how process improvement will help them (ROI) NDIA Conference - 38
Contact Information Debra J. Perry SW Engineer Division Process Group Harris Corporation debra.perry@harris.com 321-727-6830 Harris Corporation http://www.harris.com/ P.O. Box 37 Melbourne, Florida 32902-0037 NDIA Conference - 39
BACKUP SLIDES NDIA Conference - 40
References page 1 of 4 Page 9-12 Introduction to CMMI Version 1.2 http://www.sei.cmu.edu/products/courses/p44b.h tml Page 13 Profiles of Level 5 CMMI Organizations http://www.stsc.hill.af.mil/crosstalk/2007/01/0701 Reifer.html Page 15 No Satisfaction at Toyota http://www.fastcompany.com/magazine/111/ope n_no-satisfaction.html NDIA Conference - 41
References page 2 of 4 Page 20 - Performance Results of CMMI - Based Process Improvement 2006 http://www.sei.cmu.edu/pub/documents/06.repo rts/pdf/06tr004.pdf Page 21 - Lockheed Martin results http://www.sei.cmu.edu/activities/cmmi/results/p dfs/2003-cmmi-016.pdf Page 22 - Integrating PSP sm and CMMI Level 5. Gabriel Hoffman, Northrop Grumman IT. May 1, 2003 NDIA Conference - 42
References page 3 of 4 Page 24 - Case Study of an Improvement Program Featuring Reviews and Inspections, by: Robert MacFarland, Ericsson Ltd.; American Society for Quality, Software Quality Professional, June 2001, Vol. 3, No. 3 Page 25 Process Maturity Profile CMMI v1.1 SCAMPI v1.1 Class A Appraisal Results 2005 Mid-Year Update, September 2005 http://www.sei.cmu.edu/appraisalprogram/profile/pdf/cmmi/2005sepcmmi.pdf Page 26 - SEI CMMI Published Appraisal Results http://sas.sei.cmu.edu/pars/ NDIA Conference - 43
References page 3 of 4 Page 34-35 Getting There: Tips and Considerations for Marching Toward CMMI Maturity Level 4 or 5, by Gary Natwick, Debbie Perry and Sophie Boyd, SEPG Conference 26-29 March 2007 NDIA Conference - 44
Process Improvement Plan Level 3 4 5 Management Process Capability Change Process Capability Not Understood Quantitatively Stable and Controlled Optimized Process Approach Proactive Predictive Continuous Improvement Improvement Focus Project Division and Project Division and Project Measurement Focus Monitor and Control Decision Making Systemic Issues Analysis Comparative Statistical Causal Benefits Repeatable Practices Predictable Results Innovation and Improvement NDIA Conference - 45
ROI for Management Example 1 of 2 (Profiles of Level 5 CMMI Organizations) NDIA Conference - 46
ROI for Management Example 2 of 2 (Profiles of Level 5 CMMI Organizations) NDIA Conference - 47
Raytheon North TX L5 we want to achieve the performance excellence goals required by our business. We are focused on achieving performance excellence and recognition as the preferred supplier for new business. NDIA Conference - 48
Northrop Grumman L5 NDIA Conference - 49
Industry Investment In Process Maturity GCSD vs. Industry Headcount from SEI Survey (December 2004) 2.0% 1.8% 1.6% 1.4% 1.2% 1.0% 0.8% Industry Median 1.9% of Organization Northrop-Grumman (Melbourne) 1.5% of Engineering 0.6% 0.4% 0.2% 0.0% GCSD FY05 GCSD FY06 REQ GCSD FY05 GCSD FY06 REQ Where does you company fit? NDIA Conference - 50
Process Improvement Flowchart Document Processes Determine Goals Define Process Improvement Determine Measures Collect Data Analyze Data Yes Process Improved? No NDIA Conference - 51
ROI Process Flowchart Determine Scope Estimate Current Costs Estimate Change Benefits Calculate ROI Present to Management Track Actual Costs Yes Change Approved? No NDIA Conference - 52