Choosing the Right Measures - Prerequisite for Changing the Game Elliot Chikofsky, EM&I Fellow 2010 ENGINEERING, MANAGEMENT & INTEGRATION, INC. 455 Spring Park Pl.Ste.350 Herndon, VA 20170-4727 Tel: 703.742.0585 Fax: 703.742.8034 www.em-i.com
Report Documentation Page Form Approved OMB No. 0704-0188 Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. 1. REPORT DATE APR 2010 2. REPORT TYPE 3. DATES COVERED 00-00-2010 to 00-00-2010 4. TITLE AND SUBTITLE Choosing the Right Measures - Prerequisite for Changing the Game 5a. CONTRACT NUMBER 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) Engineering, Management & Integration Inc (EM&I),455 Spring Park Pl.Ste.350,Herndon,VA,20170-4727 8. PERFORMING ORGANIZATION REPORT NUMBER 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR S ACRONYM(S) 12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited 11. SPONSOR/MONITOR S REPORT NUMBER(S) 13. SUPPLEMENTARY NOTES Presented at the 22nd Systems and Software Technology Conference (SSTC), 26-29 April 2010, Salt Lake City, UT. Sponsored in part by the USAF. U.S. Government or Federal Rights License 14. ABSTRACT 15. SUBJECT TERMS 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT a. REPORT unclassified b. ABSTRACT unclassified c. THIS PAGE unclassified Same as Report (SAR) 18. NUMBER OF PAGES 23 19a. NAME OF RESPONSIBLE PERSON Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18
The Challenge How do you replace the hull and streamline the structure while ethe eship is steaming at full speed in rough waters? 2
Value, Architecture, and Performance Value Recognition Value engineering involves understanding business value and making informed IT decisions that maximize value. Enterprise architecture puts adaptive frameworks in place that enable rapid response at the speed of today s business. Enterprise Architecture 3 Performance Engineering Performance engineering means measuring position, speed, and rate of change and being ready and able to act on those measures.
Value, Architecture, and Pereformance Performance Engineering Measures Value Recognition Enterprise Architecture Business Di Drivers 4
What Should We Measure? Would you tell me, please, which way I ought to go from here? That depends a good deal on where you want to get to, said the Cat. I don t much care where-- said Alice. Then it doesn t matter which way you go, said the Cat. --so long as I get somewhere, Alice added as an explanation. Oh, you re sure to do that, said the Cat, if you only walk long enough. 5
Metrics Purpose Provide a moving snapshot of key performance indicators Current status Position relative to targets Movement toward targets Provide alerts to identify critical action areas Focus on leading, not lagging indicators Identification of expected bands of acceptable performance Communicate quantitative aspects of organizational goals in terms of targets and performance improvement 6
Why Measure? COMPARISON LOCATION DIRECTION X X X X X X PROGRESS VELOCITY RATES OF CHANGE 7
IT / Business Alignment The product of IT is NOT source code. It is improved business performance. Understanding and coordinating IT goals and measures with Business goals and measures is essential. IT processes are being managed as business processes that require cost / benefit analysis. 8
Measures Must Be Clearly Mapped Structural Complexity Training Profiles Staffing Profiles Technology Acceptance Organization, Culture, Skills Architectural Characteristics Level of Integration Level of Standardization Degree of Flexibility Quality Support Reliability 9 Architecture & Support Team Profiles Hierarchy Measures Stability Experience and Training Productivity vs. Quality Investment Business Effectiveness Business Value Customer Satisfaction Motivation IT Investment ROI Budget Control Risk Performance Profiles Portfolio Investment Technology Change Profile Technical Condition Complexity Performance Reusability Size Quality Change (Modification) Average Time to Implement Frequency of Releases Cost per Modification Rework Profiles IT Processes and Projects Cost IT Resources Staff/Project Schedule Process Quality Reliability MTTF Customer Responsiveness Time to Market Review Profiles
Meaningful Levels of Measurement Executive Production Decision View 100 Delay 80 Overhead - ROI 60 - Business Impact 40 - Price-performance 20 0 - Risk/Opportunity... 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 48 47 46 45 44 43 42 Manager Management View - Costs/Budget - Schedule/Effort/Delay - Standards - Resource Availability... Base ar 16.875 cv 1 Tm 12 * 1 Workload 3 34.05 30 980.271 Audit del Sdel svc Ssv c 3.51362 x 145.600 300 0 Delay>Svtm # d u Cst Dr wkld svtm 146 0.68 24.5 Base Acty a b mo max 1680.46 avg 980.271 s 280.077 min 392.198 SD Sv ctmbase x x0 Exp* a b mo 30.0067 26.1359 16.875 3.11928 BaseThrupu Alt max avg min SD 0.13 6.76 15 500.412 Audit A del Sdel svc Ssv c 0.35025 x 0.78250 300 0 Delay>Svtm # d u Cst Dr wkld svtm 0.78 0.35 16.9 Alt.Acty a b mo max 974.839 avg 500.412 s 25.9854 min 293.569 SD Sv ctmalt x Exp*A a b mo x0 49.5746 36.1301 16.875 12.4660 AltThruput max avg min SD Operations Operational View - Process/Activities iti - Products/Specs - Policy/Procedures - Constraints/Guides... TL Applications, Information Requests, Updates Obtain Technical Loan Applications 33.0 D A1 Completed Loan Applications Conduct Applicant Credit Check 53.0 D A2 OLTG Credit Approval Technical Conduct Approval Technical Review 132.0 D A3 Accept/Reject Loan Guarantee 131.0 D A4 Scientific/Engineering Review Team, DoD Review Team Application Acceptance Information Application Correspondence
Measurement for Management Levels Business Strategy Business Management Project Management Process Management Product Management... Risk/ Value Impact ROI Margins Opportunity... Staffing Schedule Costs Effort Progress Throughput Change Rates Size Cycle Times Queues Complexity Volatility Reliability... Productivity... Quality
Metrics Warehouse: From Measures to Strategy Size Effort Schedule Quality Level 1 People Quality per report period Level 2 Project Process # steps Product Team size, Total hrs budgeted, KSLOC, (FP), # application users Total $ budgeted # Pages document Labor rate % effort completed, cost by CAPP, $/KSLOC, $/page, Cost variance effort by CAPP, $/application (planned vs. actual) Defect cost Actual hours % Time completed, % steps completed, % products completed, Available hours Schedule variance # Reviews completed/ # Programs coded/ (planned vs. actual) Total # reviews, Total # programs, # Tests completed/ # Reqmts traced/ Total # tests, etc. Total # reqmts, etc. Defect correction # of requirement # defects per process # defects per product, rate, MTTR changes downtime, MTTF complexity Closed requests/ # Defects delivered # Defects discovered Customer satisfaction (effectiveness) by phase, (productivity) Age of open defects 12
Measures Change MANAGED OPTIMIZED Process Change Management Tech. Change Management Software Quality Management Quant. Process Management PROCESS LEVEL Defect Causal Analysis Technology Insertion Analysis Productivity REPEATABLE DEFINED Peer Reviews Software Product Engineering Integrated Software Management Training Programs Organization Process Definition Organization Process Focus PRODUCT LEVEL APPLICATION LEVEL Quality Metrics Reliability Design Volatility Complexity Performance Cost INITIAL Software Configuration Management Software Quality Assurance SW Project Tracking and Oversight Software Project Planning Requirements Management QUALITATIVE LEVEL Size Effort Defects (categorization) Cycle Time (schedule) 13
Comparative Analysis Prior Baselines Current Baseline Benchmarks Project Histories, Other Baselines 14
Choosing the Right Measures Measures must be ACTIONABLE. Identifiable contribution to Business decisions i IT decisions Measures must be practical to collect. Keep the number of metrics SMALL. Understand the roll-up of measures into measures at other management levels. 15
Cautions about Metrics Indicators, not absolutes Only as good as the supporting data Must be understood to be of value Business understanding di IT understanding Not for judging individual performance Cannot identify, explain, or predict everything Need iterative analysis from multiple viewpoints Avoid direct comparison of projects No single metric 16
Some Core Metrics a starting gpoint CHARACTERISTICS UNIT OF MEASURE Size Progress Counts of physical code Reuse Rework Effort Cost Counts of staff hours expended Rework Resource allocations Schedule Calendar dates tied to milestones, reviews and audits, deliverable products Quality Rework Counts of software problems Readiness for delivery and defects Improvement trends 17
Getting Started / Making Progress Priorities within a project: Understand d the data you are getting now Standardize the content of future measurement reports Define and collect the additional information you need for project planning and tracking Priorities within an organization: Understand historical data you already have Get consistent data from project to project Get consistent data over time 18
What should we measure, and why? 19 Secure executive sponsorship Identify critical stakeholders Conduct facilitated workshops: Business area leaders IT leaders Evaluate proposed measures Develop balanced scorecards Focus on the alignment and translation of IT and Business objectives Make it visible Own the responsibility Tailor the reward system
Lessons Learned I:. EM&/ 20 PARTNERSHIP INNOVATION RESULTS
Communication I:. EM&/ 21 PARTNERSHIP INNOVATION RESULTS
Balance I:. EM&/ 22 PARTNERSHIP INNOVATION RESULTS
For More Information Engineering Management & Integration Inc. 455 Spring Park Place, Suite 350 Herndon, VA 20170 USA +1 (703) 742-0585 www.em-i.com 23 Elliot Chikofsky +1 (781) 272-0049 EM&I 75 Lexington St Elliot.Chikofsky@em-i.com Burlington, MA 01803 Elliot.Chikofsky@dfas.mil USA 1999 META Group, Inc., Stamford, CT (203) 973-6700 www.metagroup.com