What Agile Learns from Waterfall And What Waterfall Learns from Agile Michael Mah Managing Partner QSM Associates, Inc. 75 South Church Street Pittsfield, MA 01201 413-499-0988 Fax 413-447-7322 e-mail: michael.mah@qsma.com Website: www.qsma.com Blog: www.optimalfriction.com 10/22/2008 1
10/22/2008 2
10/22/2008 3
QSM Corporate Profile Founded by Larry Putnam, international expert in software estimation. QSM Software Lifecycle Management Tools (SLIM Suite) used worldwide by Fortune 500 Clients, Federal, and State Agencies to measure, estimate, and control software development Offices in Washington DC, New England, London, Paris, Amsterdam, and Tokyo. QSM research findings and the methodology within SLIM are published in 5+ books, over 100+ published papers. 10/22/2008 4
Industry Data from the QSM SLIM-Metrics Database Spans 20+ years Large, worldwide heterogeneous database contains over 7,400+ projects Represents over 685+ million SLOC, 7+ million function points, over 600 languages, from 500+ organizations in 18 countries Adding 200 400 projects/year 10/22/2008 5
Partial List of Clients British Telecom SAP Microsoft DirecTV AT&T/BellSouth BMC Software Motorola Verizon Wireless Computer Sciences Corp Checkfree Corp IBM Global Misys Healthcare JPMorganChase Boeing Bank of New York Mellon Lockheed Martin Siemens Medical The Vanguard Group 10/22/2008 6
Project Interviews 10/22/2008 7
Whiteboard Sketch Network Management Project 10/22/2008 8
10/22/2008 9
Whiteboard Sketch e-commerce Application 10/22/2008 10
10/22/2008 11
10/22/2008 12
10/22/2008 13
Two Categories of Work Types Expert Thinking: Requiring creativity and expert problem solving, design of new products. Complex Communication: High paying jobs in design, innovation, and management of others with face-to-face interaction. Routine Cognitive Tasks: Work that follows well-defined logical rules, such as call center and data processing. Some routing software coding. Routine manual tasks: Physical labor jobs such as blue-collar assembly line work. Nonroutine manual tasks: Physical labor jobs difficult to automate requiring optical recognition, fine motor control, including a range of factory jobs. 10/22/2008 14
Software Design and Code LifeCycle People Feasibility Requirements HL Design Low Level Design Code/Unit test Integration System Test Field Trials Post-Delivery Corrections Elapsed Calendar Time Critical Domain Knowledge and Complex Communication Occurs Here 10/22/2008 15
Software Design and Code LifeCycle People Feasibility Requirements HL Design Low Level Design Code/Unit test Integration System Test Field Trials Post-Delivery Corrections Elapsed Calendar Time Lack of Domain Knowledge, Communication Breakdowns Show Up Here 10/22/2008 16
Release Retrospectives 1 - COLLECT AND VALIDATE PROJECT DATA 2 - ANALYZE PROJECTS USING QSM REFERENCE DATABASE Overview of Database Number of Projects vs Division Avg, Min, Max PI Grade vs Division Desktop Faxes 5.0 Desktop Faxes High End Systems 3.0 Division High End Systems Division Mobile Coms 3.0 Mobile Coms 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 Number of Projects Average Value of Metrics 0 5 10 15 20 25 30 35 40 Number of Projects Life Cycle Effort vs ESLOC 100 New Percent SLOC Modified % SLOC Unmodified % SLOC 12 35 56 10 1 0.1 Life Effort (MHR) (thousands) 0 5 10 15 20 25 30 35 40 45 50 55 60 65 Number of Projects 0.01 1 10 100 1000 ESLOC (thousands) ALL Systems Fax Corp 1997base Avg. 4 - DOCUMENT RESULTS 3 - DETERMINE PROCESS METRICS & PROJECT POSITIONING 10/22/2008 17
Co-Located XP - Follett Software Team size 24 Developers 7 Testers 3 Customers 3 Project Leaders Code Base 1,000,000 lines of code 7,000 automated unit test 10,000 automated acceptance test 18
People Management XP Says XP works in small to medium sized teams How we evolved or extended this rule Subteams 1 large room is mandatory Trade-offs Communication between subteams 1 room noise level (distractions) Lack of personal space 19
10/22/2008 20
10/22/2008 21
10/22/2008 22
Input to SLIM Size Defects Time Effort 10/22/2008 23
SLIM Replica Destiny 5.0 Staffing & Probability Analysis STORY BUILD Avg Staff (people) <Destiny Release 5.0> 1 3 2 4 5 6 7 8 30 25 Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9-99R 10-99.9R 20 15 10 5 Avg Staff (people) Oct '03 1 2 3 4 5 6 7 8 9 Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep '04 0 SOLUTION PANEL - <Destiny Release 5.0> Duration Effort Cost Peak Staff MTTD Start Date BUILD 6.5 151 1283.5 23.2 0.675 1/5/2004 Life Cycle 7.2 162 1377.2 23.2 0.675 12/15/2003 Months PM $ (K) people Days PI=23.1 MBI=5.6 Eff SLOC=218,531 CONTROL PANEL - <Destiny Release 5.0> 23.1 23.2 219 18.5 27.7 PI 18.5 27.8 Peak Staff 175 262 Eff SLOC (K) 10/22/2008 24
Trendline Assessment Build Phase Staffing Main Build Peak Staff vs. Size 1,000 100 Rel 6.0 Rel 7.5 Rel 7.0 Rel 8.0 Rel 5.0 Rel 6.5 10 Peak Staff (FTEs) 1 Normal Staffing 0.1 100 1,000 Effective SLOC (thousands) Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 Business Av g. Line Sty le 1 Sigma Line Sty le 10/22/2008 25
Trendline Assessment Build Phase Schedule Main Build Phase Duration vs Size 100 Rel 5.0 Rel 8.0 Rel 6.5 10 Time (Months) Rel 6.0 Rel 7.5 Rel 7.0 Schedules are Half Industry 1 100 1,000 Effective SLOC (thousands) Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 Business Av g. Line Sty le 1 Sigma Line Sty le 10/22/2008 26
Trendline Assessment Defects/Quality Defects During Test 10,000 1,000 Rel 6.0 Rel 7.5 Rel 7.0 Rel 8.0 Rel 6.5 Errors (SysInt-Del) Rel 5.0 100 Far Fewer Defects: 50% - 66% Below Industry 10 100 1,000 Effective SLOC (thousands) Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 Business Av g. Line Sty le 1 Sigma Line Sty le 10/22/2008 27
Follett vs. Industry Average Industry Average Current Performance Delta Project Cost $3.5 Million $2.2 Million -$1.3M Schedule 12.6 months 7.8 months -4.8 mos Defects During QA 242 121-50% Staffing 35 35 n/a * Using average project size of 500,000 lines of new and modified code 10/22/2008 28
Follett and XP: It has worked incredibly well Destiny Library Manager: Award of Excellence 2004, presented by Technology and Learning magazine (December 2004). Awards Portfolio 2004, presented by Media and Methods magazine (May/June 2004). Technology & Learning Award of Excellence 2006, 2007 Destiny Textbook Manager Awards Portfolio 2005, presented by Media and Methods magazine (May/June 2005). Technology & Learning Award of Excellence 2007 Destiny Enriched Services Technology & Learning Award of Excellence 2007 Follett Software provides Library Automation Solutions to 52% of the K12 market. Destiny Library Manager: Single largest product market share in K12 with 19% of the total market and continues to outpace the competition in market growth. 29
10/22/2008 30
Benchmark Comparison of Five Agile Companies Highlighted: BMC Software Distributed SCRUM 10/22/2008 31
Project Interviews 10/22/2008 32
Whiteboard Sketch Performance Mgr R2.3 10/22/2008 33
Defect Type (All) 160 Count of Severity* Release 2.3 Defect Rate 140 120 100 80 Product+* 60 40 20 0 Create Date Status Mode Status Severity* TR-Version 10/22/2008 34
Input to SLIM-DataManager Size Defects Time Effort 10/22/2008 35
SLIM Replica PerfMgr Rel 2.3 Staffing & Probability Analysis R&D C&T P_Mnt 1 Avg Staff (people) <Performance Manager Rel 2.3> 32 4 5 6 7 8 9 10 120 100 Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9-99R 10-99.9R 80 60 40 20 Avg Staff (people) Apr '06 1 2 3 4 5 6 7 8 9 May Jun Jul Aug Sep Oct Nov Dec Jan Feb '07 Mar 0 SOLUTION PANEL - <Performance Manager Rel 2.3> Duration Effort Cost Peak Staff MTTD Start Date C&T 5.3 488 4880.0 92.8 0.104 7/2/2006 Life Cycle 7.0 556 5561.2 92.8 0.232 6/1/2006 Months PM $ (K) people Days PI=28.3 MBI=8.3 Eff SLOC=844,710 CONTROL PANEL - <Performance Manager Rel 2.3> 28.3 92.8 845 22.6 33.9 PI 74.2 111.3 Peak Staff 676 1014 Eff SLOC (K) 10/22/2008 36
Agile Assessment Schedule BUILD Phase Schedule Agile projects are faster as a whole. (BMC (and also Follett) are highlighted) 100 10 C&T Duration (Months) BMC Rel 2.4 BMC Rel 2.3 1 10 100 1,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style 1 Sigma Line Style 10/22/2008 37
Agile Assessment Staffing BUILD Phase Staffing Agile Projects team sizes are fairly typical BMC elects to run with large teams. 1,000 BMC Rel 2.4 BMC Rel 2.3 100 C&T Peak Staff (People) 10 1 10 100 1,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style 1 Sigma Line Style 10/22/2008 38
Agile Assessment Quality Bugs Follett and BMC bug rates are significantly lower 10,000 1,000 BMC Rel 2.4 BMC Rel 2.3 100 Errors (SysInt-Del) 10 1 10 100 1,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style 1 Sigma Line Style 10/22/2008 39
Summary View Agile Data Main Build Trends BUILD Phase Schedule 100 BUILD Phase Ef f ort 10,000 10 C&T Duration (Months) 1,000 100 10 C&T Effort (PM) 1 10 100 1,000 STORIES (thousands) BUILD Phase Staf f ing Agile projects as a whole achieve faster speed 1,000 1 10 100 1,000 STORIES (thousands) Bugs 10,000 100 10 C&T Peak Staff (People) 1,000 100 10 Errors (SysInt-Del) 1 10 100 1,000 STORIES (thousands) Low Defects for BMC & Follett 1 10 100 1,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Av g. Line Sty le 1 Sigma Line Sty le 10/22/2008 40
Four Core Metrics: Project History Produced Software (Size) How long? How much? How good? Duration Effort Discovered Defects 10/22/2008 41
Example PI Calculation Rel 2.3 PI = SIZE TIME EFFORT = 28.3 * Size = 918 Stories/837k LOC Effort = 488 Person-Months Time = 5.25 Months (Build) 10/22/2008 42
Business Scientific Productivity Index (PI) (industry values by application type) Information System Process Control Engineering Telecommunications Command and Control Real Time Avionics Real Time Microcode 0 2 4 6 8 10 12 14 16 18 20 22 24 Productivity Index (PI) w/ ±1 Standard Deviation 10/22/2008 43
Productivity Index: Five Companies Using Agile Avg, Min, Max PI vs Organization Agile #1 - Follett BMC and Follett lead the pack Agile #2 - BMC Company B Organization Company C Company D 0 5 10 15 20 25 30 35 40 Avg, Min, Max PI All Systems Avg. Line Style 10/22/2008 44
Productivity Index Assessment Productivity Index/Velocity Follett and BMC Indexes are exemplary 35 30 25 20 PI 15 10 5 0 10 100 1,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style 1 Sigma Line Style 10/22/2008 45
Agile Assessment Quality Bugs But blue dots (other 3 companies) had higher than average defects 10,000 1,000 BMC Rel 2.4 BMC Rel 2.3 100 Errors (SysInt-Del) 10 1 10 100 1,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style 1 Sigma Line Style 10/22/2008 46
BMC vs. Industry Average Industry Average Current Performance Delta Project Cost $5.5 Million $5.2 Million -$.3M Schedule 15 months 6.3 months -8.7 mos Defects During QA 713 635-11% Staffing 40 92 +52 * Using average project size of 150,000 lines of new and modified code 10/22/2008 47
BMC Secret Sauce 10/22/2008 48
Buy-In BMC Secret Sauce (con t) VP-Level (or higher) Senior Executive Sponsorship Scrum Master Training Core Group Energized and Passionate Staying Releasable Nightly Builds/Test 2-week Iteration Demos Frequent, Rigorous Peer Code Review Dusk-to-Dawn Teamwork Communication Techniques for Information Flow Wikis, Video-conferencing, Periodic On-Site Meetings Co-Located Release Planning Scrum of Scrum Meetings (US Time) 10/22/2008 49
Backlogs BMC Secret Sauce (con t) One Master Backlog AND Multiple Backlog Management One Setup for User Stories Across Teams Added Requirements Architect to Interface Product Mgt with R&D Holding Back the Waterfall Test Driven Development Retrospective Meetings to Not Regress into old Waterfall Habits Outside Source to Audit the Process 10/22/2008 50
Benchmark Comparison of Offshore Waterfall Development 10/22/2008 51
A CXO Comment Chunk it, routinize it, digitize it, and then send it offshore for an army of people offshore to code it 10/22/2008 52
Staffing View Outsourced Project IN C EP ELAB CONST TRA N S 0 Avg Staff (people) <Overall September Baseline> 1 3 4 5 6 8 9 10 120 100 80 60 Avg Staff (people) 40 20 Apr '03 1 3 5 7 9 11 13 15 17 19 21 Jun Aug Oct Dec Feb Apr Jun Aug Oct Dec Feb '04 '05 0 10/22/2008 53
Defects vs. Industry Average 0 1 Defects Found 3 4 5 5 6 6 7 8 7 8 9 9 10 10 350 300 250 200 150 Defects 100 50 05/17 '03 6 12 18 24 30 36 42 48 54 60 66 72 78 84 90 06/28 08/09 09/20 11/01 12/13 01/24 03/06 04/17 05/29 07/10 08/21 10/02 11/13 12/25 02/05 '04 '05 0 Current Plan Actuals Green Control Bound Yellow Control Bound Project: CEP 10/22/2008 54
Industry Average vs. Offshore Industry Average Offshore Average Delta Project Cost $3.5 Million $3.2 Million -$0.2M Schedule 12.3 months 9.6 months -2.7 mos Cumulative Defects 2,702 7,565 +280% Staffing 33 50 +17 * Using average project size of 150,000 lines of new and modified code 10/22/2008 55
It s Not Easy to Blend Minds Across the Planet 10/22/2008 56
10/22/2008 57
Models Used for this Analysis SLIM- Estimate Size, Schedule, Cost & Quality Estimating SLIM- Metrics Metrics Repository & Analysis In-Flight Forecasting SLIM- Control 10/22/2008 58
www.qsma.com 10/22/2008 59
Recommended Reading Mah, Michael, How Agile Measures Up, and What It Means Agile Advisory Executive Report Vol 9 Number 9. 2008 Cutter Information Corp. Reassessing XP, by Tom DeMarco/Cutter Business Technology Council, Business Technology Trends Advisory, Volume 6, No. 8. Cutter Information Corp. Putnam, Lawrence H., and Myers, Ware, Five Core Metrics, The Intelligence Behind Successful Software Management 2003 Dorset House Publishers. 10/22/2008 60
Contact: For Additional Information Michael Mah Benchmark Practice Director, Cutter Consortium Managing Partner, QSM Associates michael.mah@qsma.com Tel: (413) 499-0988 x103 Website: www.qsma.com Blog: www.optimalfriction.com 10/22/2008 61