Integrated Measurements for CMMI

Transcription

1 Integrated Measurements for CMMI Gary Natwick Harris Corporation Gary Natwick - 1 Where are we? Melbourne, Florida Gary Natwick - 2 1

2 Government Communications Systems Division $1.8B in Sales 8,000 Employees ISO 9001:2000 CMMI Level 3 DoD s Civil s National s Strategic Management and Business Development Homeland Security s Harris Technical Services Corporation Gary Natwick - 3 Process Improvement At GCSD CMMI Level / Process SW-CMM Level 4 Compliance Monitor (PCM) /04 6/ Lean Initiated Integrated Process Manual () 4/04 1/ Division ISO 9001:2000 Process Group (DPG) 1/04 7/ ISO 2000 SEI 9001 Compliance Partner Network / CMMI 4/01 9/ % Reduction 225% Improvement SW-CMM Level 3 7/ in Defects in Productivity 1996 SW-CMM Level 2 1/ Project Engineering Metrics (PEM) 1/ SW-CMM SW-CMM CMMI 1992 Level 3 Level 4 Level 3 Engineering Process Group (EPG) /95 Software Software Engineering Process Group (SEPG) 7/ Gary Natwick - 4 2

3 Why We Measure Characterize Gain understanding of integrated es, products, and resources Establish baselines for future comparisons Evaluate Measurement indicators show when projects and es are drifting off track, so they can brought back under control Assess achievement of quality goals and impacts of technology and improvements on products and es Predict Predictive measures are also the basis for trending, so estimates for cost, time, and quality can be updated based on current evidence Gain an understanding of relationships among es and products for future prediction Improve Identify roadblocks, root causes, inefficiencies, and other opportunities for improving product quality and performance Measures of current performance to compare against and judge whether or not improvement actions are working as intended and what the side effects may be Gary Natwick - 5 Goal-Driven Measurements Project Management Planning, estimating, monitoring, and controlling project: costs, schedules and quality Process Improvement Providing baseline data, measuring trends, tracking root causes of problems and defects, and identifying and implementing changes for improvement Organizational Vision Effectively applying unified end-to-end integrated es and methods encompassing proven and emerging standards/approaches for the purpose of delivering high-quality cost competitive system solutions to our customers Gary Natwick - 6 3

4 Goal-Question-Metric (GQM) Business Goals What are our business goals? Improve customer satisfaction by reducing defects Measurement Goals What do we want to achieve in order to satisfy our business goals? Reduce post-delivery defects to N per KLOC G1 G2 Questions Metric What questions will help us plan & manage progress toward our goals? Where are defects introduced & removed? How effective are peer reviews? What metrics are necessary to answer these questions? Defects detected in peer reviews, testing... Defect categorization, rework time... Q1 Q2 Q3 M1 M2 M3 M4 The question is not: What metrics should I use? Rather: What do I want to know or learn? Why are we collecting the data? How do we use the data? Adapted from: Goal-Driven Software Measurement - A Guidebook, Park et al., CMU/SEI-96-HB-002, August next level solutions Gary Natwick - 7 Integrated Measurement Process Planning Metrics used to support quantitative management Planned and/or expected performance in the metrics including any required goals and/or control limits Variance implication and corrective action for metrics falling outside control limits Source and collection mechanism of the measurement data Responsible persons for collection, analysis, reporting, and managing Collection Occurs at the periodic intervals defined in the project plans and is monitored for completeness, integrity and accuracy Primary source for actual data is in the accounting systems used to manage the project (e.g., financial management, configuration management, change management, risk management) Data is input into the division standard metric tool each period Gary Natwick - 8 4

5 Integrated Measurement Process Analysis Metrics are communicated graphically for a clear and easily understood message Better to have many graphs than it is to have many messages on one graph Metrics are indicators that give warnings of problems associated with issues An issue may be tracked with several metrics using different measures Trend-based metrics when expected or planned values change regularly over time to determine whether the performance implied is achievable Limit-based metrics when expected or planned values remain relatively constant over time to determine whether the performance crosses its established bounds Reporting Quantitative management decisions are communicated to project team members, management and customers Integrated into the management and occurs as soon as possible after analysis has been completed to assure that there is time for corrective action Metric s falling outside the control limits are reviewed and corrective actions are recorded and tracked to closure Gary Natwick - 9 Integrated Engineering Metrics Frames display one or more metrics Metrics are composed of measures Control Panels display one or more frames metric1 metric1 metric2 metric2 metric3 metric1 metric2 metric3 metric1 metric2 metric3 metric1 metric2 metric3 metric1 metric2 metric3 Metric 1 Metric 2 metric3 Metric 3 Reports contain one or more control panels time Measures are units of measurement measure1 measure1 measure2 measure2 measure3 measure1 measure2 measure3 measure1 measure2 measure3 measure1 measure2 measure1 measure2 measure3 Measure 1 Measure 2measure3 Measure 3 time Gary Natwick

6 Electrical Engineering Pins Issues/Actions "Pins" issues/actions Electrical Engineering Gates Issues/Actions "Gates" issues/actions Project Engineering Issues/Actions CSF 1 - HW setup delayed CSF 2 - SW understaffed CSF 3 - Requirements changes CSF 4 - Work Interruption CSF 5 - Power Outages CSF 6 - Fire Drills CSF 7 - Build Problems TPM 1 Notes TPM 2 Notes TPM 3 Notes Etc, Etc ME Issues/Actions/Comments mechanical issues/actions KEY TPMS PARAMETER SPEC CURRENT TREND Vehicle Weight 1560 lbs 1560 lbs Y Power LRU # KW 1.37 KW R Power LRU # KW 1.23 KW G C-Band Xmit SNR 92 db 93 db Y KU -Band Output Power 2kw 1.68 KW R CSCI #1 Memory Util 75% 12% G CSCI #1 CPU Utilization 50% 20% G CSCI #2 Memory Util 75% 66% Y CSCI #2 CPU Utilization 50% 67% R FPGA Gate Util 75% 88% R Frames Limits Trending Over Time Current/Previous Period Data Gary Natwick - 11 Integrated Project Compliance Performance Progress Cost and Schedule Resources Software Performance Electrical Performance Mechanical Performance System I&T Performance Peer Reviews Management Gary Natwick

7 Engineering Performance Gary Natwick - 13 Engineering Progress Gary Natwick

8 Engineering Cost & Schedule Gary Natwick - 15 Engineering Resources Gary Natwick

9 Software Performance Gary Natwick - 17 Software Defect Category (Zoom) Gary Natwick

10 Electrical Engineering Performance Electrical Engineering Pins Issues/Actions "Pins" issues/actions Electrical Engineering Gates Issues/Actions "Gates" issues/actions Gary Natwick - 19 Mechanical Engineering Performance ME Issues/Actions/Comments mechanical issues/actions Gary Natwick

11 System I&T Gary Natwick - 21 Peer Reviews Gary Natwick

12 Management Review KEY TPMS PARAMETER SPEC Vehicle Weight 1560 lbs Power LRU # KW Power LRU # KW C-Band Xmit SNR 92 db KU -Band Output Power 2kw CSCI #1 Memory Util 75% CSCI #1 CPU Utilization 50% CSCI #2 Memory Util 75% CSCI #2 CPU Utilization 50% FPGA Gate Util 75% CURRENT 1560 lbs 1.37 KW 1.23 KW 93 db 1.68 KW 12% 20% 66% 67% 88% TREND Y R G Y R G G Y R R Project Engineering Issues/Actions CSF 1 - HW setup delayed CSF 2 - SW understaffed CSF 3 - Requirements changes CSF 4 - Work Interruption CSF 5 - Power Outages CSF 6 - Fire Drills CSF 7 - Build Problems TPM 1 Notes TPM 2 Notes TPM 3 Notes Etc, Etc Gary Natwick - 23 Measuring Process Compliance Integrated Process Foundation Organizational requirements Process Model compliance (CMMI ) Integration and collaboration across functional organizations Disciplined repeatable es with objective criteria Entry/exit criteria, inputs, outputs, verification, measures Planning each, and tracking against plan Tailoring standard es and assets Budgets, schedules, resources Managing established baselines Managing Stakeholder involvement Measuring progress and improvement Gary Natwick

13 Integrated Process Manual Management Processes Life-Cycle Processes Support Processes Organizational Processes Planning Estimation Monitoring and Control Supplier Acquisition & Management Change Management Proposal Development Requirements Analysis System Architecting/Design Design Code and Unit Test Fabrication and Assembly Product Integration Verification Validation Production Field Support Requirements Management Risk Management Configuration and Data Management Metrics Decision Analysis and Resolution Peer Review Design Review Quality Assurance Integrated Logistics Support Process Improvement Training Division Metrics Gary Natwick - 25 Management Processes Customer Requirements / RFP All Processes All Processes Plans Status Corrective Action Establish Life-Cycle es Technical Plan Monitoring & Control Supplier Product Monitor Re-plan Scope of work Establish Supplier Acquisition & Management Planning Contract Scope of work Estimates Supplier Agreement Historical Data and Models Estimation Change Management Division Metrics Change Gary Natwick

14 Life-Cycle Processes - 1 Proposal Development Requirements Analysis System Architecting/ Design Design Fab/Assembly Code and Unit Test Product Integration Verification Process Validation Process Startup Review Life-Cycle Phase Business Acquisition System Requirements System Design Prelim Design Detail Design Fab, Code, Integration Verification Proposal Requirements Functional Allocated Design Developmental Configuration Product Milestones / Reviews TBR PCR SRR SDR PDR CDR TRR System Test PCA, FCA Key Products Proposal Prog Plans Sys Arch Prelim Design Prog Plans (P) Requirements Sys Design Detail Design Sys Arch (P) CONOPS Interface Defn Design docs Operational Threads / Use Cases Technical Data Package Traceability Test cases / descriptions Traceability Assembled Components Component test procs / results Integration plan (F) Integration procedures Integration results Test procedures Test results Traceability Delivered systems Gary Natwick - 27 Life-Cycle Processes - 2 Life-Cycle Phase Milestones / Reviews Key Products Production Field Support Verification Validation Other Life-Cycle es (as applicable) Production Field Support Product Production Readiness Review Production plan Delivered systems As-built documents Test results Product Site Transition / Install Plan Revisions to product baseline Test results Production and Field Support es apply only to the extent required by contract May be not applicable May implement revisions to the baseline products May involve other life cycle es Requirements, design, implementation Production Process Produce and deliver multiple systems Field Support Process Site installation Operations support Engineering services Gary Natwick

15 Support Processes Requirements Change Requests Requirements Management Decision Analysis & Resolution Peer Review Design Review Requirements Impact Issues Structured Decisions Products Identified Defects Products Approved Risk Management Status Actions Management es Life-Cycle es Identify Monitor Manage Info Needs Metrics Metrics Analyses Products Changes s Configuration and Data Management Product Product Support Data Integrated Logistics Support Processes Products Compliance Issues Quality Assurance Gary Natwick - 29 Organizational Processes Organizational Processes Division Objectives Process Improvement Division Metrics Training Standard Historical metrics Process assets Trained staff metrics assets Lessons learned Tailoring Plan Defined Process Management es Life-Cycle es Support es Gary Natwick

16 99 MONTH 9999 < TEMPLATE FOR PMP THAT IS DELIVERED TO MEET A PMP CDRL REQT > Contract Number: XXXXXXXXXXXXXXXXX Prepared for: CUSTOMER ADDRESS CITY-STATE-ZIP Prepared by: HARRIS CORPORATION Government Communications Systems Division P.O. Box 37 Melbourne, FL USA Integrated Compliance Approach Projects XYZ Management Plan CDRL XXXX Document Control Number: CMMI Tailoring Process Compliance Evidence Improve Reuse Submit Command Media Organization Organizational Learning Historical Data Best Practices Example Assets Improvement Requests Process Compliance Metric Gary Natwick - 31 Process Compliance Evidence Overview A brief description of the intent Entry Criteria State, Prerequisites, Criteria Inputs Needed work products, resources Required Activities Mandatory tasks to implement the Measures Process performance against plans Verification Process compliance oversight Exit Criteria State, Criteria Outputs Resulting work products Organizational Improvement Information Metrics, reusable work products Tailoring Approved tailoring, specific Implementation Guidance Common implementation descriptions evidence needed to demonstrate compliance Supporting Documentation and Assets Applicable organizational references Gary Natwick

17 Process Compliance Scores ASSESSMENT STATUS COLORS NY NA NS Not Yet Not Applicable Not Scored To be appraised at a later date (i.e., the has not yet been executed by the and cannot be appraised) Not applicable to the project (e.g., Code and Unit Test Process is not applicable to a production-type program) Pending an appraisal PROCESS COMPLIANCE COLORS FI LI PI NI Fully Implemented Largely Implemented Partially Implemented Not Implemented Direct artifacts are present and appropriate No substantial weaknesses Direct artifacts are present and appropriate One or more substantial weaknesses Direct artifact is absent or inadequate Substantiated by indirect artifact/affirmation One or more substantial weaknesses Any situation not covered by the above Gary Natwick - 33 Process Compliance Measures Represents overall compliance score for program Based on lowest color score harsh, but in keeping with CMMI standards Depicts scoring distribution over all items More insight on the overall project score (A) A B Depicts score for each executed or being executed by this program 3 columns identify types of es In PCM, point+click on underlined acronym drills down to scoring details for the C Gary Natwick

18 Lessons Learned - 1 One metric doesn t tell the whole story Need an integrated and many times orthogonal views Trending is key Project planning is key Data collection is the hardest Having standard tools is highly desirable Consistency User friendly Easy access Training is a must Cultural change is hard Train everything, even the obvious Gary Natwick - 35 Lessons Learned - 2 Engineering Measurement Handbook - Goals - Processes - Management Plan Project Engineering Metrics Tool - Projects - Pre-defined Metrics - Measurement Specifics Measurement Information Model Information Need Section Indicator Section Derived Measure Specification Section Base Measure Specification Section Attribute Section Information Need Information Category Measurable Concept Indicator Analysis Model Decision Criteria Derived Measure Measurement Function Base Measures Measurement Method Type of Method Scale Type of Scale Unit of Measurement Relevant Entities Attributes Not Just Engineering Enterprise Measurement Tool - SPC - User Defined & Controlled - Integrated Disciplines & Databases Gary Natwick

19 Contact Information Gary Natwick SEI-Authorized CMMI Instructor SEI-Authorized SCAMPI SM Lead Appraiser SEI-Authorized SCAMPI SM B&C Team Leader Harris SEI Partner Business & Technical Point of Contact Harris Corporation P.O. Box 37 Melbourne, Florida Licensed by the Software Engineering Institute (SEI) to provide: SEI Introduction to CMMI courses SEI SCAMPI SM Appraisal Services Capability Maturity Model Integration, CMMI, and CMM are registered with the U.S. Patent and Trademark Office. SCAMPI is a service mark of Carnegie Mellon University. Gary Natwick