Quantifying the Impact

Transcription

1 Quantifying the Impact Of Agile Practices! #RallyON13! Larry

2 Choose wisely Warning: Leave now (take the blue pill) if you are not prepared to have your most closely held beliefs challenged with data.

3 LARRY MACCHERONE

4 You know that a Lean and Agile approach is better; now we are extracting the numbers to say HOW MUCH BETTER. Giving you the tools you need to acquire resources from management and motivate your teams to accomplish (sometimes expensive and/ or painful) change.

5 Use the Software Development Performance Index (SDPI) to: Gain feedback Make better decisions Increase your performance How is this going to help me?

6 If you implement a lean/agile measurement regime the traditional way, you will fail. when you cannot express it in numbers, your What gets knowledge is of a meagre measured gets and unsatisfactory kind done. -Lord Kelvin

7 þ Feedback measures are used to improve your own performance. Lever measures are employed to change someone else s behavior. Metrics as feedback NOT levers (Sin #1)

8 Too often we seek knowledge not wisdom. As a result, we are becoming a nation of technological giants and ethical infants. ~Vernon Davis and Omar Bradley Measurement should Process change is complement, NOT replace qualitative insight (sin #3)

9 Measurement Insight Decision Outcome Strategy fails one small decision at a time ODIM the virtue for sin #6

10 The Seven Deadly Sins of Agile Measurement Deadly Sin 1 Using metrics as levers to change someone else s behavior Heavenly Virtue Using metrics for feedback to improve your own performance 2 Unbalanced metrics Day-one have one metric from each quadrant 3 Believing metrics can replace thinking Use quantitative insight to complement rather than replace qualitative insight 4 Too costly metrics Favor automatic metrics from passively acquired data or lightweight surveys 5 Using a convenient metric Use ODIM to determine metrics the provide critical insight and drive to your desired outcomes 6 Using bad analysis Get your statistics right by consulting experts 7 Forecasting without discussing probability Use the percentile coverage distribution, the cone of uncertainty, or Monte Carlo simulation

11 Larry Maccherone Mark Smith Michael Dellanoce Kevin Chabreck Jennifer Maccherone (photo by CC license from Sarah Fagg) Red Pill Team

12 1. Validate a balanced set of performance measures 2. Quantify the correlation between decisions and these outcome measures 3. Provide a framework to help you implement a healthy metrics regime within your organization. 4. Prototype quantitative feedback features (dashboards, balanced scorecards, benchmarking, etc.). Red Pill mission

13 What have we done? Crawled around in an anonymous copy of all users of Rally Numbers but no text Extracted metrics of performance (outcomes) Extracted (via heuristics) decisions made Identified the correlation between decisions and performance This talk is the first public airing of these findings

14 Unique teams for this research Organization Size Teams % < 50 Seats % Seats % Seats % Seats % > 1000 Seats % Total: %

15 Progression of reaction to contrary evidence 1. The finding must be wrong or not nuanced enough. I can ignore it. 2. The presenter must have an agenda. I should call BS. 3. Maybe I should question my bias.

16 Caveats

17 Strategy games Each decision slightly affects your chances

18 Statistically significant but NOT predictive yet Also, correlation does not equal causation

19 The Software Development Performance Index (SDPI) 4, 6, or 7 dimensions Extracted from ALM/ Kanban tool Responsiveness Quality Predictability Productivity Acquired via survey Customer satisfaction Employee engagement Acquired from hit-rate/hotspot analytics Build-the-right-thing metric

20 Responsiveness Time In Process (TiP) similar to cycle/lead time In Process varies by work item type, context, etc. Quality Mostly Released Defects Density (team size normalized) Also: DRE, Cumulative Defect Aging, etc. Predictability Stability (lack of variation) of Throughput Say/Do (pending) Productivity Throughput / team size Meaning of SDPI dimensions

21 Decision: Estimating approach

22 TaskEstimateTotal above 0 for > 50% of Stories* PlanEstimate above 0 for > 50% of Stories* Iteration on > 90% of Stories * where ScheduleState >= In-Progress Determining process type

23 Which do you think performs the best? Vote Process Type 1 Iterations w/o estimates (Iteration) 2 Full Scrum (Iteration + PlanEstimate + TaskEstimateTotal) 3 Lightweight Scrum (Iteration + PlanEstimate) 4 Hourly-oriented Iterative (Iteration + TaskEstimateTotal) Teams 3% 79% 10% 8%

24 Estimating with points and/or task hours Points and task hours (Full Scrum) second best overall Points only (Lightweight Scrum) better overall 1 Iterative No estimates 2 Full Scrum 3 Lightweight Scrum 4 Hour-oriented

25 Estimating with points and/or task hours Points and task hours (Full Scrum) second best except for Quality Points only (Lightweight Scrum) better overall TiP (lower is better) Defect density (lower is better) Variability of Throughput (lower is better) Throughput (higher is better)

26 It s OK to compare Throughput

27 Decision: Dedicate to one team Most consistently good Agile recommendation

28 Dedicating folks to one team Much better: Predictability and Productivity Better: Quality; Mildly better: Responsiveness TiP (lower is better) Defect density (lower is better) Variability of Throughput (lower is better) Throughput (higher is better)

29 Dedicating to one team: The most followed recommendation Keep the teams stable: The least followed recommendation 1 out of every 4 team members change every 3 months

30 From one quarter to the next, the median team is 74.4% stable. 1 out of 4 team members will change every 3 months. Teams are surprisingly unstable

31 Stability Much better: Predictability and Productivity Mildly better: Responsiveness and Quality TiP (lower is better) Defect density (lower is better) Variability of Throughput (lower is better) Throughput (higher is better)

32 Decision: Work in Process (WiP)

33 The most obvious finding Empirical confirmation of Little s Law

34 The most dramatic finding 4x swing in Quality from lowest to highest WIP

35 Lower WIP Much better: Quality and Responsiveness Not so much: Predictability and Productivity TiP (lower is better) Defect density (lower is better) Variability of Throughput (lower is better) Throughput (higher is better)

36 A huge puzzle

37 Median generally tracks the mean

38 Why does the median not track the mean for WiP to TiP?

39 Median tracks mean in TiP for other conditions

40 Organization size impacts decisions Those decisions affect performance But organization size ALONE does not strongly affect performance (with a few unexplained exceptions) Does organization size matter? Yes and no

41 Other conclusions (no charts in the interest of time) Team Size 5-9 out performs 1-3 and 3-5 However, larger teams seem to perform just as well Scrum vs. Kanban Scrum 1.25x-3x better Defect Density (Quality) Scrum slightly better on Predictability Productivity AND RESPONSIVENESS ScrumBan is the best of both worlds (except for Quality)

42 Mythbusters Myth Responsiveness Quality Predictability Productivity Points + hours better than points alone Dedicate to one team Keep the teams stable Lower WIP is always better Ideal team size: 5-9 Kanban is better than Scrum ScrumBan is the best of both worlds Busted Confirmed Minimal difference

43 SDPI upgrades Paper publish Product Balanced Scorecards Dashboards Benchmarking Looking forward

44 Findings will change All decision metrics are heuristics (later surveys) We re constantly improving the heuristics (or getting new survey results) Just last week, we were saying that Team Stability was not that big of a factor Scrum versus Kanban results are different from previously reported Data science research is 90% cleaning the data

45 How might I use the SDPI?

46 More decisions to correlate Team makeup decisions Ratio of various roles Co-location Process decisions Ceremonies, feedback mechanisms used, etc. Engineering practices decisions TDD, continuous integration, etc. Context Age of codebase, industry, etc. Send me a comment on SpotMe or to participate in the survey Show under what context certain findings hold Decisions survey going out

47 What do I do now? Agree to fill out a decisions survey and allow us to correlate it with your team(s) metrics ( ) Your name/company will not be published/ discussed Use the Software Development Performance Index (SDPI) at your business Send me your hypotheses (twitter) Send me a comment on SpotMe or to participate in the survey Early access to folks whose teams fill out the research survey

48 A fact without a theory Is like a ship without a sail, Is like a boat without a rudder, Is like a kite without a tail. A fact without a figure Is a tragic final act. But one thing worse In this universe Is a theory without a fact. ~George Schultz Larry LMaccherone@rallydev.com Please use your Ipad/ SpotMe to review this keynote Thanks!

49 LARRY MACCHERONE