Is Agile Keeping Its Promises? Time To Market, Productivity & Quality

Transcription

1 Is Agile Keeping Its Promises? What Industry Metrics Show On Time To Market, Productivity & Quality Michael Mah Managing Partner QSM Associates, Inc. 75 South Church Street Pittsfield, MA USA Website: Blog: Andrea Gelli Managing Director QSM Associates Switzerland 8032 Zürich M andrea.gelli@qsma.ch Website:

2 Michael Mah Michael Mah is the director of the Benchmarking Practice at the Cutter Consortium, a Boston-based IT think-tank, and served as past editor of the IT Metrics Strategies publication. He is also managing partner at QSM Associates Inc. based in Massachusetts USA. Michael teaches, writes, and consults to technology companies on measuring, estimating and managing software projects, whether in-house, offshore, waterfall, or agile. With over 25 years of experience, Michael and his partners have derived productivity patterns for thousands of software projects collected worldwide across engineering and business applications. His current work examines time-pressure dynamics of teams, and its role in project success and failure. In addition to his background in physics and electrical engineering, he is a mediator specializing in dispute resolution for technology projects. He has a degree in electrics engineering from Tufts University. His training on dispute resolution, mediation, and participatory processes is from the Program on Negotiation at Harvard Law School and the Radcliffe Institute for Advanced Study. He can be reached at michael.mah@qsma.com. (#2)

3 Who We Are: QSM maintains the world s largest benchmarking database of 10,000+ completed software projects collected worldwide. We put industry productivity statistics on the desktop. The QSM SLIM database contains projects in all industries, waterfall, Agile, offshore/outsourced, in-house, new development, and maintenance. SLIM tools enable managers to measure and estimate Agile and/or waterfall projects, and determine ROI. (#3)

4 Partial List of Clients British Telecom SAP Microsoft Intel AT&T/BellSouth BMC Software Motorola VerizonWireless Roche Diagnostics Fiserv Corp IBM Global Misys Healthcare JPMorganChase Boeing Bank of New York Mellon Lockheed Martin Progressive Insurance DirecTV (#4)

5 (#5)

6 (#6)

7 (#7) Agile Asks: Are You Getting Lean?

8 Co-Located XP - Follett Software Team size 24 Developers 7 Testers 3 Customers 3 Project Leaders Code Base 1,000,000 lines of code 7,000 automated unit test 10,000 automated acceptance test (#8)

9 (#9)

10 (#10)

11 (#11)

12 (#12)

13 (#13)

14 (#14)

15 Agile Release Retrospectives 1 - COLLECT AND VALIDATE PROJECT DATA 2 - ANALYZE PROJECTS USING QSM REFERENCE DATABASE (#15) 4 - PRESENT RESULTS 3 - DETERMINE PROCESS METRICS & PROJECT POSITIONING

16 Four Core Metrics: Your History Produced Software (Size) How long? How much? How good? Duration Effort Discovered Defects (#16)

17 (#17) Project Sketch Core Metrics

18 Input to SLIM Size Defects Time Effort (#18)

19 SLIM Model Replica Destiny 5.0 Staffing & Probability Analysis STORY BUILD How Does This Release Compare Against Industry Data? Avg Staff (people) <Destiny Release 5.0> Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9-99R R Avg Staff (people e) 5 Oct ' Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep '04 0 SOLUTION PANEL - <Destiny Release 5.0> Duration Effort Cost Peak Staff MTTD Start Date BUILD /5/2004 Life Cycle /15/2003 Months PM $ (K) people Days PI=23.1 MBI=5.6 Eff SLOC=218,531 CONTROL PANEL - <Destiny Release 5.0> PI Peak Staff Eff SLOC (K) (#19)

20 Follett vs. Industry Average Industry Average Current Performance Delta Project Cost $3.5 Million $2.2 Million -$1.3M Schedule 12.6 months 7.8 months -4.8 mos QA Defects % Staffing n/a (#20)

21 Offshore vs. Industry Average Industry Average Offshore Average Delta Project Cost $3.5 Million $3.2 Million -$0.3M Schedule 12.6 months 9.6 months -3.0 mos QA Defects % Staffing (#21) * Normalized to same project size

22 Trendline Assessment Build Phase Schedule Main Build Phase Duration vs Size 100 Rel 5.0 Rel 8.0 Rel Time (Months) Rel 6.0 Rel 7.5 Rel 7.0 Schedules are Half Industry ,000 Effective SLOC (thousands) (#22) Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 Business Av g. Line Sty le 1 Sigma Line Sty le

23 Trendline Assessment Build Phase Staffing Main Build Peak Staff vs. Size 1, Rel 6.0 Rel 7.5 Rel 7.0 Rel 8.0 Rel 5.0 Rel Peak Staff (FTEs) 1 Normal Staffing ,000 Effective SLOC (thousands) (#23) Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 Business Av g. Line Sty le 1 Sigma Line Sty le

24 Trendline Assessment Defects/Quality Defects During Test 10,000 1,000 Rel 6.0 Rel 7.5 Rel 7.0 Rel 8.0 Rel 6.5 Error ors (SysInt-Del) Rel Far Fewer Defects: 50% - 66% Below Industry ,000 Effective SLOC (thousands) (#24) Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 Business Av g. Line Sty le 1 Sigma Line Sty le

25 Follett and XP: It has worked incredibly well Destiny Library Manager: Award of Excellence 2004, presented by Technology and Learning magazine (December 2004). Awards Portfolio 2004, presented by Media and Methods magazine (May/June 2004). Technology & Learning Award of Excellence 2006, 2007 Destiny Textbook Manager Awards Portfolio 2005, presented by Media and Methods magazine (May/June 2005). Technology & Learning Award of Excellence 2007 Destiny Enriched Services Technology & Learning Award of Excellence 2007 Follett Software provides Library Automation Solutions to 52% of the K12 market. Destiny Library Manager: Single largest product market share in K12 with 19% of the total market and continues to outpace the competition in market growth. (#25)

26 (#26) Increased REVENUES

27 Domain Knowledge Smart people, experienced people Coding is moving knowledge from mind into the machine Inexperience costs money (#27)

28 Short Feedback Loops Paired programmers Instantaneous code reviews Accelerated learning and execution Face to face communication channel (#28)

29 Time Boxing Short iterations Clear and discernible progress Anticipation of the next important feature Efficiency (#29)

30 Avoiding Burnout XP = Sustainable pace 40 Hour Work Weeks Prevent productivity collapse for overworked teams (#30)

31 Craftsmanship Over Craft Take pride in what you do Do not compromise professionalism Simple design Upfront testing Prevent costly rework Build it right the first time (#31)

32 Transparency Transparence is a great floodlight. People who thrive in political maneuvering hate SCRUM - Ken Schwaber (#32)

33 High-bandwidth Communication The best teams have wide-open pipes Domain knowledge moves among the team Information flows rapidly and accurately (#33)

34 Avoiding Waste and Costly Rework Rework has high cost Rework takes time Rework creates defects Rework is bad Refactoring can be a cover up (#34)

35 (#35) Distributed SCRUM BMC Software

36 (#36) Whiteboard Sketch Performance Mgr R2.3

37 Defect Type (All) Release 2.3 Defect Rate 160 Count of Severity* Product+* (#37) Create Date Status Mode Status Severity* TR-Version

38 Input to SLIM-Model Size Defects Time Effort (#38)

39 SLIM Model Replica Rel 2.3 Staffing & Probability Analysis R&D C&T How Does This Release P_Mnt 1 Compare Against Industry Data? Avg Staff (people) <Performance Manager Rel 2.3> Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9-99R R Avg Staff (people) Apr ' May Jun Jul Aug Sep Oct Nov Dec Jan Feb '07 Mar 0 SOLUTION PANEL - <Performance Manager Rel 2.3> Duration Effort Cost Peak Staff MTTD Start Date C&T /2/2006 Life Cycle /1/2006 Months PM $ (K) people Days PI=28.3 MBI=8.3 Eff SLOC=844,710 CONTROL PANEL - <Perform ance Manager Rel 2.3> PI Peak Staff Eff SLOC (K) (#39)

40 BMC vs. Industry Average Industry Average Current Performance Delta Project Cost $5.5 Million $5.2 Million -$.3M Schedule 15 months 6.3 months -8.7 mos QA Defects % Staffing (#40)

41 Agile Assessment Staffing BUILD Phase Staffing Agile Projects team sizes are fairly typical BMC (red) uses large teams. Follett (green) uses average sized teams. 1,000 BMC Rel 2.4 BMC Rel C&T Peak Staff (People) 10 Larger Teams ,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style 1 Sigma Line Style (#41)

42 Agile Assessment Schedule BUILD Phase Schedule BMC and Follett projects are very fast C&T Duration (Months) BMC Rel 2.4 BMC Rel 2.3 Faster Schedules ,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style 1 Sigma Line Style (#42)

43 Agile Assessment Quality Bugs BMC and Follett Bugs are low 10,000 1,000 BMC Rel 2.4 BMC Rel Erro rors (SysInt-Del) 10 Fewer Defects ,000 STORIES (thousands) Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style 1 Sigma Line Style (#43)

44 (#44) BMC Secret Sauce

45 BMC Secret Sauce (con t) Buy-In VP-Level (or higher) Senior Executive Sponsorship Scrum Master Training Core Group Energized and Passionate Staying Releasable Nightly Builds/Test 2-week Iteration Demos Frequent, Rigorous Peer Code Review Dusk-to-Dawn Teamwork Communication Techniques for Information Flow Wikis, Video-conferencing, Periodic On-Site Meetings Co-Located Release Planning Scrum of Scrum Meetings (US Time) (#45)

46 BMC Secret Sauce (con t) Backlogs One Master Backlog AND Multiple Backlog Management One Setup for User Stories Across Teams Added Requirements Architect to Interface Product Mgt with R&D Holding Back the Waterfall Test Driven Development Retrospective Meetings to Not Regress into old Waterfall Habits Outside Source to Audit the Process (#46)

47 (#47) Agile at Quick Solutions, Inc.

48 (#48)

49 (#49)

50 (#50)

51 (#51)

52 (#52)

53 Agile Assessment Staffing Average Staff 1,000 Smaller Teams 100 Headcount ,000 New + Modified Size (thousands) (#53) All Systems Special Project QSM 2008 Business Avg. Line Style 1 Sigma Line Style

54 Agile Assessment Schedule Time-to-Market 100 Faster Schedules 10 Months ,000 New + Modified Size (thousands) (#54) All Systems Special Project QSM 2008 Business Avg. Line Style 1 Sigma Line Style

55 Agile Assessment Quality Bugs During QA 10,000 Fewer Defects 1, Defects ,000 New + Modified Size (thousands) (#55) All Systems Special Project QSM 2008 Business Avg. Line Style 1 Sigma Line Style

56 Curve Fitting and Creating Agile Trends Iterations/Build Phase Trends Time-to-Market 100 Effort 10,000 1, Months Person-Months ,000 New + Modified Size (thousands) ,000 New + Modified Size (thousands) Average Staff 1,000 Bugs During QA 10, Headcount 1, Defects ,000 New + Modified Size (thousands) ,000 New + Modified Size (thousands) (#56) All Sy stems QSM 2008 Business Av g. Line Sty le 1 Sigma Line Sty le

57 Curve Fitting and Creating Agile Trends Agile Trends - Iterations/Build Phase Time-to-Market 100 Effort 10,000 1, Months Person-Months ,000 New + Modified Size (thousands) ,000 New + Modified Size (thousands) Average Staff 1,000 Bugs During QA 10, Headcount 1, Defects ,000 New + Modified Size (thousands) ,000 New + Modified Size (thousands) (#57) All Sy stems Av g. Line Sty le 1 Sigma Line Sty le

58 New Agile Benchmark Trends Agile Trends - Iterations/Build Phase Time-to-Market 100 Effort 10,000 1, Months Person-Months ,000 New + Modified Size (thousands) ,000 New + Modified Size (thousands) Average Staff 1,000 Bugs During QA 10, Headcount 1, Defects ,000 New + Modified Size (thousands) ,000 New + Modified Size (thousands) (#58)

59 (#59) Software - Is It Possible That The Earth Is Growing A Global Brain?

60 (#60)

61 (#61)

62 (#62)

63 (#63)

64 Who We Are: QSM maintains the world s largest benchmarking database of 10,000+ completed software projects collected worldwide. We put industry productivity statistics on the desktop. The QSM SLIM database contains projects in all industries, waterfall, Agile, offshore/outsourced, in-house, new development, and maintenance. SLIM tools enable managers to measure and estimate Agile and/or waterfall projects, and determine ROI. (#64)

65 Partial List of Clients British Telecom SAP Microsoft Intel AT&T/BellSouth BMC Software Motorola VerizonWireless Roche Diagnostics Fiserv Corp IBM Global Misys Healthcare JPMorganChase Boeing Bank of NY Mellon Lockheed Martin Progressive Insurance DirecTV (#65)

66 SLIM- Estimate: Size, Schedule, Cost & Quality Estimating SLIM- MasterPlan: Incremental Development & Project SLIM-Control: Aggregation Variance Analysis & Adaptive Forecasting SLIM-Metrics: Industry Benchmarking & Process Improvement SLIM-DataManager Software Project Metrics Repository (#66) footer

67 For Additional Information Michael Mah website: blog: Tel: Andrea Gelli andrea.gelli@qsma.ch Website: Tel: (#67)