The Good, the Bad and the Puzzling: The Agile Experience at 5 Companies (Updated!) Michael Mah Managing Partner QSM Associates, Inc. 75 South Church Street Pittsfield, MA 01201 413-499-0988 Fax 413-447-7322 e-mail: michael.mah@qsma.com Website: www.qsma.com Blog: www.optimalfriction.com
Are You Getting Lean?
Co-Located XP - Follett Software Team size 24 Developers 7 Testers 3 Customers 3 Project Leaders Code Base 1,000,000 lines of code 7,000 automated unit test 10,000 automated acceptance test 5
SLIM Replica Destiny 5.0 Staffing & Probability Analysis STORY Avg Staff (people) How BUILD Does This Release<Destiny Release 5.0> 1 3 2 4 Compare Against Industry Data? 5 6 7 8 30 25 Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9-99R 10-99.9R 20 15 10 Avg Staff (people) 5 Oct '03 1 2 3 4 5 6 7 8 9 Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep '04 0 SOLUTION PANEL - <Destiny Release 5.0> Duration Effort Cost Peak Staff MTTD Start Date BUILD 6.5 151 1283.5 23.2 0.675 1/5/2004 Life Cycle 7.2 162 1377.2 23.2 0.675 12/15/2003 Months PM $ (K) people Days PI=23.1 MBI=5.6 Eff SLOC=218,531 CONTROL PANEL - <Destiny Release 5.0> 23.1 23.2 219 18.5 27.7 PI 18.5 27.8 Peak Staff 175 262 Eff SLOC (K)
Follett vs. Industry Average Industry Average Current Performance Delta Project Cost $3.5 Million $2.2 Million -$1.3M Schedule 12.6 months 7.8 months -4.8 mos QA Defects 242 121-50% Staffing 35 35 n/a
Trendline Assessment Build Phase Schedule Main Build Phase Duration vs Size 100 Rel 5.0 Rel 8.0 Rel 6.5 10 Time (Months) Rel 6.0 Rel 7.5 Rel 7.0 Schedules are Half Industry 1 100 1,000 Effective SLOC (thousands) Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 Business Av g. Line Sty le 1 Sigma Line Sty le
Trendline Assessment Defects/Quality Defects During Test 10,000 1,000 Rel 6.0 Rel 7.5 Rel 7.0 Rel 8.0 Rel 6.5 Errors (SysInt-Del) Rel 5.0 100 Far Fewer Defects: 50% - 66% Below Industry 10 100 1,000 Effective SLOC (thousands) Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 Business Av g. Line Sty le 1 Sigma Line Sty le
Follett and XP: It has worked incredibly well Destiny Library Manager: Award of Excellence 2004, presented by Technology and Learning magazine (December 2004). Awards Portfolio 2004, presented by Media and Methods magazine (May/June 2004). Technology & Learning Award of Excellence 2006, 2007 Destiny Textbook Manager Awards Portfolio 2005, presented by Media and Methods magazine (May/June 2005). Technology & Learning Award of Excellence 2007 Destiny Enriched Services Technology & Learning Award of Excellence 2007 Follett Software provides Library Automation Solutions to 52% of the K12 market. Destiny Library Manager: Single largest product market share in K12 with 19% of the total market and continues to outpace the competition in market growth. 12
Domain Knowledge Smart people, experienced people Coding is moving knowledge from mind into the machine Inexperience costs money
Short Feedback Loops Paired programmers Instantaneous code reviews Accelerated learning and execution Face to face communication channel
Short iterations Clear and discernible progress Anticipation of the next important feature Efficiency Time Boxing
Avoiding Burnout XP = Sustainable pace 40 Hour Work Weeks Prevent productivity collapse for overworked teams
Take pride in what you do Do not compromise professionalism Simple design Upfront testing Prevent costly rework Build it right the first time Craftsmanship Over Craft Measure Twice, Cut Once (says Norm Abram)
Transparency The best teams hide nothing Daily stand-ups Iteration Demos Scrum sessions Velocity and burn-down charts
High-bandwidth Communication The best teams have wide-open pipes Domain knowledge moves among the team Information flows rapidly and accurately
Avoiding Waste and Costly Rework Rework has high cost Rework takes time Rework creates defects Rework is bad Refactoring can be a cover up
Distributed SCRUM BMC Software
Whiteboard Sketch Performance Mgr R2.3
Input to SLIM-DataManager Size Defects Time Effort
SLIM Replica Rel 2.3 Staffing & Probability Analysis R&D C&T How Does This Release P_Mnt 1 Compare Against Industry Data? Avg Staff (people) <Performance Manager Rel 2.3> 3 2 4 5 6 7 8 9 10 120 100 Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9-99R 10-99.9R 80 60 40 20 Avg Staff (people) Apr '06 1 2 3 4 5 6 7 8 9 May Jun Jul Aug Sep Oct Nov Dec Jan Feb '07 Mar 0 SOLUTION PANEL - <Performance Manager Rel 2.3> Duration Effort Cost Peak Staff MTTD Start Date C&T 5.3 488 4880.0 92.8 0.104 7/2/2006 Life Cycle 7.0 556 5561.2 92.8 0.232 6/1/2006 Months PM $ (K) people Days PI=28.3 MBI=8.3 Eff SLOC=844,710 CONTROL PANEL - <Perform ance Manager Rel 2.3> 28.3 92.8 845 22.6 33.9 PI 74.2 111.3 Peak Staff 676 1014 Eff SLOC (K)
BMC vs. Industry Average Industry Average Current Performance Delta Project Cost $5.5 Million $5.2 Million -$.3M Schedule 15 months 6.3 months -8.7 mos QA Defects 713 635-11% Staffing 40 92 +52
BMC Secret Sauce
Buy-In BMC Secret Sauce (con t) VP-Level (or higher) Senior Executive Sponsorship Scrum Master Training Core Group Energized and Passionate Staying Releasable Nightly Builds/Test 2-week Iteration Demos Frequent, Rigorous Peer Code Review Dusk-to-Dawn Teamwork Communication Techniques for Information Flow Wikis, Video-conferencing, Periodic On-Site Meetings Co-Located Release Planning Scrum of Scrum Meetings (US Time)
Backlogs BMC Secret Sauce (con t) One Master Backlog AND Multiple Backlog Management One Setup for User Stories Across Teams Added Requirements Architect to Interface Product Mgt with R&D Holding Back the Waterfall Test Driven Development Retrospective Meetings to Not Regress into old Waterfall Habits Outside Source to Audit the Process
Agile Release Retrospectives 1 - COLLECT AND VALIDATE PROJECT DATA 2 - ANALYZE PROJECTS USING QSM REFERENCE DATABASE Overview of Database Number of Projects vs Division Avg, Min, Max PI Grade vs Division Desktop Faxes 5.0 Desktop Faxes High End Systems 3.0 Division High End Systems Division Mobile Coms 3.0 Mobile Coms 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 Number of Projects Average Value of Metrics 0 5 10 15 20 25 30 35 40 Number of Projects Life Cycle Effort vs ESLOC 100 New Percent SLOC 56 Modified % SLOC 12 Unmodified % SLOC 35 0 5 10 15 20 25 30 35 40 45 50 55 60 65 Number of Projects 1 10 100 1000 ESLOC (thousands) 10 1 0.1 0.01 Life Effort (MHR) (thousands) ALL Systems Fax Corp 1997base Avg. 4 - DOCUMENT RESULTS 3 - DETERMINE PROCESS METRICS & PROJECT POSITIONING
Four Core Metrics: Your History Produced Software (Size) How long? How much? How good? Duration Effort Discovered Defects
Agile Assessment Staffing BUILD Phase Staffing Agile Projects team sizes are fairly typical BMC (red) uses large teams. Follett (green) uses average sized teams. 1,000 BMC Rel 2.4 BMC Rel 2.3 100 C&T Peak Staff (People) 10 Larger Teams 1 10 100 1,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style 1 Sigma Line Style
Agile Assessment Schedule BUILD Phase Schedule BMC and Follett projects are very fast 100 10 C&T Duration (Months) BMC Rel 2.4 BMC Rel 2.3 Faster Schedules 1 10 100 1,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style 1 Sigma Line Style
Agile Assessment Quality Bugs BMC and Follett Bugs are low 10,000 1,000 BMC Rel 2.4 BMC Rel 2.3 100 Errors (SysInt-Del) 10 Fewer Defects 1 10 100 1,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style 1 Sigma Line Style
The Agile Impact Report: Time-to-Market and Productivity Trends
Time-to-Market Trend Time-to-Market 100 Faster Schedules Inovis 10 Months Homeaway BMC CNET Moody's Accuro 1 10 100 1,000 New + Modified Size (thousands) Agile Companies Rally Projects QSM 2005 Business Avg. Line Style 1 Sigma Line Style
Productivity Trend QSM Productivity Index 35 BMC 30 Accuro Moody's Homeaway Inovis 25 20 CNET PI 15 10 Higher Productivity 5 0 10 100 1,000 New + Modified Size (thousands) Agile Companies Rally Projects QSM 2005 Business Avg. Line Style 1 Sigma Line Style
Agile Database vs. Industry Trends
Curve Fitting and Creating Agile Trends
New Agile Benchmark Trends
www.qsma.com
Who We Are: QSM maintains the world s largest benchmarking database of 8,000+ completed software projects collected worldwide. We put industry productivity statistics on the desktop. The QSM SLIM database contains projects in all industries, waterfall, Agile, offshore/outsourced, in-house, new development, and maintenance. SLIM tools enable managers to measure and estimate Agile and/or waterfall projects, and determine ROI.
Partial List of Clients British Telecom SAP Microsoft Intel AT&T/BellSouth BMC Software Motorola VerizonWireless Progressive Insurance Allstate IBM DirecTV JPMorganChase Boeing Bank of New York Mellon Lockheed Martin Siemens Medical Vanguard Group
SLIM- Estimate: Size, Schedule, Cost & Quality Estimating The SLIM-Suite SLIM- MasterPlan: Incremental Development & Project Aggregation SLIM-Control: Variance Analysis & Adaptive Forecasting The SLIM-Suite SLIM-Metrics: Industry Benchmarking & Process Improvement SLIM-DataManager Software Project Metrics Repository
For Additional Information Michael Mah Managing Partner michael.mah@qsma.com Tel: 413-499-0988 x103 website: www.qsma.com blog: www.optimalfriction.com