What Agile Learns from Waterfall And What Waterfall Learns from Agile

Transcription

1 What Agile Learns from Waterfall And What Waterfall Learns from Agile Michael Mah Managing Partner QSM Associates, Inc. 75 South Church Street Pittsfield, MA Fax Website: Blog: 10/22/2008 1

2 10/22/2008 2

3 10/22/2008 3

4 QSM Corporate Profile Founded by Larry Putnam, international expert in software estimation. QSM Software Lifecycle Management Tools (SLIM Suite) used worldwide by Fortune 500 Clients, Federal, and State Agencies to measure, estimate, and control software development Offices in Washington DC, New England, London, Paris, Amsterdam, and Tokyo. QSM research findings and the methodology within SLIM are published in 5+ books, over 100+ published papers. 10/22/2008 4

5 Industry Data from the QSM SLIM-Metrics Database Spans 20+ years Large, worldwide heterogeneous database contains over 7,400+ projects Represents over 685+ million SLOC, 7+ million function points, over 600 languages, from 500+ organizations in 18 countries Adding projects/year 10/22/2008 5

6 Partial List of Clients British Telecom SAP Microsoft DirecTV AT&T/BellSouth BMC Software Motorola Verizon Wireless Computer Sciences Corp Checkfree Corp IBM Global Misys Healthcare JPMorganChase Boeing Bank of New York Mellon Lockheed Martin Siemens Medical The Vanguard Group 10/22/2008 6

7 Project Interviews 10/22/2008 7

8 Whiteboard Sketch Network Management Project 10/22/2008 8

9 10/22/2008 9

10 Whiteboard Sketch e-commerce Application 10/22/

11 10/22/

12 10/22/

13 10/22/

14 Two Categories of Work Types Expert Thinking: Requiring creativity and expert problem solving, design of new products. Complex Communication: High paying jobs in design, innovation, and management of others with face-to-face interaction. Routine Cognitive Tasks: Work that follows well-defined logical rules, such as call center and data processing. Some routing software coding. Routine manual tasks: Physical labor jobs such as blue-collar assembly line work. Nonroutine manual tasks: Physical labor jobs difficult to automate requiring optical recognition, fine motor control, including a range of factory jobs. 10/22/

15 Software Design and Code LifeCycle People Feasibility Requirements HL Design Low Level Design Code/Unit test Integration System Test Field Trials Post-Delivery Corrections Elapsed Calendar Time Critical Domain Knowledge and Complex Communication Occurs Here 10/22/

16 Software Design and Code LifeCycle People Feasibility Requirements HL Design Low Level Design Code/Unit test Integration System Test Field Trials Post-Delivery Corrections Elapsed Calendar Time Lack of Domain Knowledge, Communication Breakdowns Show Up Here 10/22/

17 Release Retrospectives 1 - COLLECT AND VALIDATE PROJECT DATA 2 - ANALYZE PROJECTS USING QSM REFERENCE DATABASE Overview of Database Number of Projects vs Division Avg, Min, Max PI Grade vs Division Desktop Faxes 5.0 Desktop Faxes High End Systems 3.0 Division High End Systems Division Mobile Coms 3.0 Mobile Coms Number of Projects Average Value of Metrics Number of Projects Life Cycle Effort vs ESLOC 100 New Percent SLOC Modified % SLOC Unmodified % SLOC Life Effort (MHR) (thousands) Number of Projects ESLOC (thousands) ALL Systems Fax Corp 1997base Avg. 4 - DOCUMENT RESULTS 3 - DETERMINE PROCESS METRICS & PROJECT POSITIONING 10/22/

18 Co-Located XP - Follett Software Team size 24 Developers 7 Testers 3 Customers 3 Project Leaders Code Base 1,000,000 lines of code 7,000 automated unit test 10,000 automated acceptance test 18

19 People Management XP Says XP works in small to medium sized teams How we evolved or extended this rule Subteams 1 large room is mandatory Trade-offs Communication between subteams 1 room noise level (distractions) Lack of personal space 19

20 10/22/

21 10/22/

22 10/22/

23 Input to SLIM Size Defects Time Effort 10/22/

24 SLIM Replica Destiny 5.0 Staffing & Probability Analysis STORY BUILD Avg Staff (people) <Destiny Release 5.0> Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9-99R R Avg Staff (people) Oct ' Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep '04 0 SOLUTION PANEL - <Destiny Release 5.0> Duration Effort Cost Peak Staff MTTD Start Date BUILD /5/2004 Life Cycle /15/2003 Months PM $ (K) people Days PI=23.1 MBI=5.6 Eff SLOC=218,531 CONTROL PANEL - <Destiny Release 5.0> PI Peak Staff Eff SLOC (K) 10/22/

25 Trendline Assessment Build Phase Staffing Main Build Peak Staff vs. Size 1, Rel 6.0 Rel 7.5 Rel 7.0 Rel 8.0 Rel 5.0 Rel Peak Staff (FTEs) 1 Normal Staffing ,000 Effective SLOC (thousands) Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 Business Av g. Line Sty le 1 Sigma Line Sty le 10/22/

26 Trendline Assessment Build Phase Schedule Main Build Phase Duration vs Size 100 Rel 5.0 Rel 8.0 Rel Time (Months) Rel 6.0 Rel 7.5 Rel 7.0 Schedules are Half Industry ,000 Effective SLOC (thousands) Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 Business Av g. Line Sty le 1 Sigma Line Sty le 10/22/

27 Trendline Assessment Defects/Quality Defects During Test 10,000 1,000 Rel 6.0 Rel 7.5 Rel 7.0 Rel 8.0 Rel 6.5 Errors (SysInt-Del) Rel Far Fewer Defects: 50% - 66% Below Industry ,000 Effective SLOC (thousands) Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 Business Av g. Line Sty le 1 Sigma Line Sty le 10/22/

28 Follett vs. Industry Average Industry Average Current Performance Delta Project Cost $3.5 Million $2.2 Million -$1.3M Schedule 12.6 months 7.8 months -4.8 mos Defects During QA % Staffing n/a * Using average project size of 500,000 lines of new and modified code 10/22/

29 Follett and XP: It has worked incredibly well Destiny Library Manager: Award of Excellence 2004, presented by Technology and Learning magazine (December 2004). Awards Portfolio 2004, presented by Media and Methods magazine (May/June 2004). Technology & Learning Award of Excellence 2006, 2007 Destiny Textbook Manager Awards Portfolio 2005, presented by Media and Methods magazine (May/June 2005). Technology & Learning Award of Excellence 2007 Destiny Enriched Services Technology & Learning Award of Excellence 2007 Follett Software provides Library Automation Solutions to 52% of the K12 market. Destiny Library Manager: Single largest product market share in K12 with 19% of the total market and continues to outpace the competition in market growth. 29

30 10/22/

31 Benchmark Comparison of Five Agile Companies Highlighted: BMC Software Distributed SCRUM 10/22/

32 Project Interviews 10/22/

33 Whiteboard Sketch Performance Mgr R2.3 10/22/

34 Defect Type (All) 160 Count of Severity* Release 2.3 Defect Rate Product+* Create Date Status Mode Status Severity* TR-Version 10/22/

35 Input to SLIM-DataManager Size Defects Time Effort 10/22/

36 SLIM Replica PerfMgr Rel 2.3 Staffing & Probability Analysis R&D C&T P_Mnt 1 Avg Staff (people) <Performance Manager Rel 2.3> Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9-99R R Avg Staff (people) Apr ' May Jun Jul Aug Sep Oct Nov Dec Jan Feb '07 Mar 0 SOLUTION PANEL - <Performance Manager Rel 2.3> Duration Effort Cost Peak Staff MTTD Start Date C&T /2/2006 Life Cycle /1/2006 Months PM $ (K) people Days PI=28.3 MBI=8.3 Eff SLOC=844,710 CONTROL PANEL - <Performance Manager Rel 2.3> PI Peak Staff Eff SLOC (K) 10/22/

37 Agile Assessment Schedule BUILD Phase Schedule Agile projects are faster as a whole. (BMC (and also Follett) are highlighted) C&T Duration (Months) BMC Rel 2.4 BMC Rel ,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style 1 Sigma Line Style 10/22/

38 Agile Assessment Staffing BUILD Phase Staffing Agile Projects team sizes are fairly typical BMC elects to run with large teams. 1,000 BMC Rel 2.4 BMC Rel C&T Peak Staff (People) ,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style 1 Sigma Line Style 10/22/

39 Agile Assessment Quality Bugs Follett and BMC bug rates are significantly lower 10,000 1,000 BMC Rel 2.4 BMC Rel Errors (SysInt-Del) ,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style 1 Sigma Line Style 10/22/

40 Summary View Agile Data Main Build Trends BUILD Phase Schedule 100 BUILD Phase Ef f ort 10, C&T Duration (Months) 1, C&T Effort (PM) ,000 STORIES (thousands) BUILD Phase Staf f ing Agile projects as a whole achieve faster speed 1, ,000 STORIES (thousands) Bugs 10, C&T Peak Staff (People) 1, Errors (SysInt-Del) ,000 STORIES (thousands) Low Defects for BMC & Follett ,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Av g. Line Sty le 1 Sigma Line Sty le 10/22/

41 Four Core Metrics: Project History Produced Software (Size) How long? How much? How good? Duration Effort Discovered Defects 10/22/

42 Example PI Calculation Rel 2.3 PI = SIZE TIME EFFORT = 28.3 * Size = 918 Stories/837k LOC Effort = 488 Person-Months Time = 5.25 Months (Build) 10/22/

43 Business Scientific Productivity Index (PI) (industry values by application type) Information System Process Control Engineering Telecommunications Command and Control Real Time Avionics Real Time Microcode Productivity Index (PI) w/ ±1 Standard Deviation 10/22/

44 Productivity Index: Five Companies Using Agile Avg, Min, Max PI vs Organization Agile #1 - Follett BMC and Follett lead the pack Agile #2 - BMC Company B Organization Company C Company D Avg, Min, Max PI All Systems Avg. Line Style 10/22/

45 Productivity Index Assessment Productivity Index/Velocity Follett and BMC Indexes are exemplary PI ,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style 1 Sigma Line Style 10/22/

46 Agile Assessment Quality Bugs But blue dots (other 3 companies) had higher than average defects 10,000 1,000 BMC Rel 2.4 BMC Rel Errors (SysInt-Del) ,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style 1 Sigma Line Style 10/22/

47 BMC vs. Industry Average Industry Average Current Performance Delta Project Cost $5.5 Million $5.2 Million -$.3M Schedule 15 months 6.3 months -8.7 mos Defects During QA % Staffing * Using average project size of 150,000 lines of new and modified code 10/22/

48 BMC Secret Sauce 10/22/

49 Buy-In BMC Secret Sauce (con t) VP-Level (or higher) Senior Executive Sponsorship Scrum Master Training Core Group Energized and Passionate Staying Releasable Nightly Builds/Test 2-week Iteration Demos Frequent, Rigorous Peer Code Review Dusk-to-Dawn Teamwork Communication Techniques for Information Flow Wikis, Video-conferencing, Periodic On-Site Meetings Co-Located Release Planning Scrum of Scrum Meetings (US Time) 10/22/

50 Backlogs BMC Secret Sauce (con t) One Master Backlog AND Multiple Backlog Management One Setup for User Stories Across Teams Added Requirements Architect to Interface Product Mgt with R&D Holding Back the Waterfall Test Driven Development Retrospective Meetings to Not Regress into old Waterfall Habits Outside Source to Audit the Process 10/22/

51 Benchmark Comparison of Offshore Waterfall Development 10/22/

52 A CXO Comment Chunk it, routinize it, digitize it, and then send it offshore for an army of people offshore to code it 10/22/

53 Staffing View Outsourced Project IN C EP ELAB CONST TRA N S 0 Avg Staff (people) <Overall September Baseline> Avg Staff (people) Apr ' Jun Aug Oct Dec Feb Apr Jun Aug Oct Dec Feb '04 ' /22/

54 Defects vs. Industry Average 0 1 Defects Found Defects /17 ' /28 08/09 09/20 11/01 12/13 01/24 03/06 04/17 05/29 07/10 08/21 10/02 11/13 12/25 02/05 '04 '05 0 Current Plan Actuals Green Control Bound Yellow Control Bound Project: CEP 10/22/

55 Industry Average vs. Offshore Industry Average Offshore Average Delta Project Cost $3.5 Million $3.2 Million -$0.2M Schedule 12.3 months 9.6 months -2.7 mos Cumulative Defects 2,702 7, % Staffing * Using average project size of 150,000 lines of new and modified code 10/22/

56 It s Not Easy to Blend Minds Across the Planet 10/22/

57 10/22/

58 Models Used for this Analysis SLIM- Estimate Size, Schedule, Cost & Quality Estimating SLIM- Metrics Metrics Repository & Analysis In-Flight Forecasting SLIM- Control 10/22/

59 10/22/

60 Recommended Reading Mah, Michael, How Agile Measures Up, and What It Means Agile Advisory Executive Report Vol 9 Number Cutter Information Corp. Reassessing XP, by Tom DeMarco/Cutter Business Technology Council, Business Technology Trends Advisory, Volume 6, No. 8. Cutter Information Corp. Putnam, Lawrence H., and Myers, Ware, Five Core Metrics, The Intelligence Behind Successful Software Management 2003 Dorset House Publishers. 10/22/

61 Contact: For Additional Information Michael Mah Benchmark Practice Director, Cutter Consortium Managing Partner, QSM Associates Tel: (413) x103 Website: Blog: 10/22/