Cycle Time Forecasting. Fast #NoEstimate Forecasting

Transcription

1 Cycle Time Forecasting Fast #NoEstimate Forecasting

2 15 th August 2013 Forecasts are attempts to answer questions about future events $1,234,000 Staff 2 Commercial in confidence

3 We estimated every task for project x and have determined it will take days Oh, over 3 months, so we ll do Project Y instead Perfect information is rarely sought or needed 3 Commercial in confidence

4 Estimate or #NoEstimate? Prioritized by Opinion / Mandatory Team Exists & Fixed in size? Date Fixed or New Team? No Estimate! 70% Chance Forecast Team Size/Skillset Options you need to Economic Prioritization Used? Forecast Date Fixed or Calculated New Team? (You just Value performed / Cost of a probabilistic forecast! Delay? 3.5 cases out of 5) Team Fixed in size? Foggy Value / Cost of Delay? Foggy Estimate GUESS! Forecast Team Size/Skillset Options Forecast Date, Manage Risks

5 Economic Prioritization same time, different value Value eg. Cost of Not Doing Time Rem. Build Time/Effort Complete Order? Product 1 3 Product 2 2 Product 3 1

6 Economic Prioritization same value different time Value eg. Cost of Not Doing Time Rem. Build Time/Effort Complete Order? Product 1 1 Product 2 2 Product 3 3

7 Suggested Economic Prioritization Cost of Delay W.S.J.F. = Prioritization Heuristic to optimize cashflow Do Highest First Duration or Time Delayed (often, the remaining time to complete) Learn this here You are here to learn this

8 8 Commercial in confidence

9 There is NO single forecast result Uncertainty In = Uncertainty Out There will always be many possible results, some more likely and this is the key to proper forecasting

10 Likelihood Probabilistic Forecasting combines many uncertain inputs to find many possible outcomes, and what outcomes are more likely than others 50% Possible Outcomes 50% Possible Outcomes Time to Complete Backlog 10 Commercial in confidence

11 Likelihood Probabilistic Forecasting combines many uncertain inputs to find many possible outcomes, and what outcomes are more likely than others 85% Possible Outcomes 15% Time to Complete Backlog 11 Commercial in confidence

12 How certain based on model forecast Further calculations to make economic tradeoffs 12 Commercial in confidence

13 Staff Dev Cost Cost of Delay Decision Induced Uncertainty Planned / Due Date Actual Date 13 July August September October November December Forecast Completion Date Commercial in confidence

14 How I Present Cycle Time Forecasts When will my feature ship? Current - 1 Devs Description Team 5 Jul 29 Jul Feature 1 25 Jul 16 Aug Feature 2 3 Aug 9 Sep Feature 3 9 Sep 4 Oct Feature 4 2 Oct 2 Nov Feature 5 10 Nov 7 Jan Feature 6 8 Jan 8 Mar Feature 7 14 What are my staffing options? Project Staffing Options Full time staff Current team Dev Staff Test Staff All Features Complete > 85% CI Mar Jan Tester Dec Devs Oct Tester Oct-2013

15 What is modelling and how to use cycle time MODELING AND CYCLE TIME 15 Use with attribution

16 A model is a tool used to mimic a real world process Models are tools for low-cost experimentation

17 Success Hinges on Being Wrong Early Initial Goal To enable decisions of staff size, cost and date for prioritization Accuracy: Eventual duration within the forecast set Next Goal To mimic the real system accurately enough to perform reliable experiments (sensitivity testing) To give the earliest warning of actual versus plan diff Early action have bigger impact Accuracy: Getting better all the time (matching actual) 17 Commercial in confidence

18 Diagnostic Simple Depth of Forecasting models Linear Projection System Cycle Time Partitioned Cycle Time Simulated process 18 Commercial in confidence

19 Simple Cycle Time Model Amount of Work (# stories) Cycle Time Random Chance / Risk 19 Commercial in confidence Parallel Work in Proc. (WIP)

20 Cycle Time = Date Complete Date Started If you have a captured start and complete date you can calculate cycle time. Note: Cycle time sometimes referred to as Lead Time but not going to discuss this now! 20 Use with attribution

21 6. Phases 1. Historical Cycle Time 2. Planned Resources/ Effort Monte Carlo Analysis = Process to Combine Multiple Uncertain Measurements / Estimates 4. Historical Scope Creep Rate (optional) The Work (Backlog) Backlog Feature 1 Feature 2 Feature 3 Commercial in confidence 5. Historical Defect Rate and Cycle Times (optional)

22 Historical Story Lead Time Trend Sum Random Numbers More often Less often Sum: Basic Cycle Time Forecast Monte Carlo Process 1. Gather historical story lead-times 2. Build a set of random numbers based on pattern 3. Sum a random number for each remaining story to build a potential outcome 4. Repeat many times to find the likelihood (odds) to build a pattern of likelihood outcomes Total Days = Sum ( Story n Random n ) Effort Days To Complete

23 23 Commercial in confidence

24 24 Commercial in confidence

25 FORECASTING STRATEGIES 25 Commercial in confidence

26 When you have historical data 1. Model Baseline using historically known truths 2. Test Model against historically known truths 3. Forecast The Past The Future

27 Compare Model vs Actual Often Number of Completed Features (Model vs Actual) Start Month 1 Month 2 Month 3 Month 4 Month 5 Model Actual

28 Forecast Trend Report Card Watch GREAT Watch Ok Bad Watch WORRY Watch Cycle Time Trend versus Model ACTIVE WIP Actual vs Model Story Count Trend versus Model Arrow Legend: Green: Heading in beneficial direction Red: Heading in damaging direction For Watch the WIP and story count trends need to be analyzed. They are 1:1 ratio of compensating importance, and may offset each other. 28 Commercial in confidence

29 When you have no historical data The Future

30 Simple Cycle Time Model Cycle Time (range guess or samples) Amount of Work (# stories, scope creep, defects) Parallel Work in Proc. (WIP on board) 31 Commercial in confidence

31 If we understand how cycle time is statistically distributed, then an initial guess of maximum allows an inference to be made Alternatives - Borrow a similar project s data Borrow industry data Fake it until you make it (AKA guess range) 33 Use with attribution

32 Probability Density Function.32 Note: Histogram from actual data x Histogram Gamma (3P) Lognormal Rayleigh Weibull 34 Commercial in confidence

33 Exponential Distribution (Weibull shape = 1) The person who gets the work can complete the work Teams with no external dependencies Teams doing repetitive work E.g. DevOps, Database teams, 35 Commercial in confidence

34 Weibull Distribution (shape = 1.5) Typical dev team ranges between 1.2 and Commercial in confidence

35 Rayleigh Distribution (Weibull shape = 2) Teams with MANY external dependencies Teams that have many delays and re-work. E.g. Test teams 37 Commercial in confidence

36 Probability Density Function Shape How Fat the distribution. 1.5 is a good starting point f(x) Location The Lower Bound Scale How Wide in Range. Related to the Upper Bound. *Rough* Guess: (High Low) / x Histogram Weibull 38 Commercial in confidence

37 What Distribution To Use... No Data at All, or Less than < 11 Samples (why 11?) Uniform Range with Boundaries Guessed (safest) Weibull Range with Boundaries Guessed (likely) 11 to 30 Samples Uniform Range with Boundaries at 5 th and 95 th CI Weibull Range with Boundaries at 5 th and 95 th CI More than 30 Samples Use historical data as bootstrap reference Curve Fitting software 39 Commercial in confidence

38 Questions Download the slides (soon) and software at Contact me Read: 40

39 Why traditional forecasting practices fail to deliver and why this is the only outcome possible CURRENT FORECASTING PRACTICES 41 Commercial in confidence

40 Development Time 5 Days (~ 15%) Total Story Lead Time 30 days Blocked and Waiting Time 9 Days Testing Time 2 Days Waiting Time 3 Days Defect Rework 2 Days Waiting Time 8 Days Release / DevOps Time 1 Day

41 Development Time 5 Days (~ 15%) Total Story Lead Time 30 days Blocked and Waiting Time 9 Days Testing Time 2 Days Waiting Time 3 Days Defect Rework 2 Days Waiting Time 8 Days Release / DevOps Time 1 Day

42 Pre Work 30 days Story / Feature Inception 5 Days Waiting in Backlog 25 days Total Story Lead Time Active Development 30 days 30 days Post Work 10 days Waiting for Release Window 5 Days System Regression Testing & Staging 5 Days

43 Pre Work 30 days Story / Feature Inception 5 Days Waiting in Backlog 25 days Total Story Lead Time 30 days Post Work 10 days 9 days (70 total) approx 13% Waiting for Release Window 5 Days System Regression Testing & Staging 5 Days Active Development 30 days

44 How to use historic data to estimate the different distributions ESTIMATING DISTRIBUTIONS USING HISTORIC DATA 48 Commercial in confidence

45 What Distribution To Use... No Data at All, or Less than < 11 Samples (why 11?) Uniform Range with Boundaries Guessed (safest) Weibull Range with Boundaries Guessed (likely) 11 to 50 Samples Uniform Range with Boundaries at 5 th and 95 th CI Weibull Range with Boundaries at 5 th and 95 th CI Bootstapping (Random Sampling with Replacement) More than 100 Samples Use historical data at random without replacement Curve Fitting 49 Commercial in confidence

46 Tools EasyFit from $499 R Free Statistics feature of KanbanSim Free Excel of course Monte Carlo features of KanbanSim $ Google for Monte Carlo Simulation Tools Oracle, Palisade, HubbardResearch (he may be here, so a big recommendation!) 50 Commercial in confidence

47 Sampling at Random Strategies If you pick what samples to use, you bias the prediction Strategies for proper random sampling Use something you know is random (dice, darts) Pick two groups using your chosen technique and compute your prediction separately and compare Don t pre-filter to remove outliers Don t sort the data, in fact randomize more if possible 51 Commercial in confidence

48 Estimating Concurrent Effort from Cumulative Flow Chart Concurrent WIP Sample : Find the smallest and the biggest or take at least 11 samples to be 90% sure of range 52 Commercial in confidence

49 20% 40% 60% 80%

50 Scope Creep Over Time Look at the rate new scope is added over time 55 Commercial in confidence

51 What is the likelihood the next sample I take will be within the range of previous samples PREDICTION INTERVALS

52 Actual Maximum Q. What is the chance of the 4 th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random) Actual Minimum

53 Actual Maximum Highest sample 2 Q. What is the chance of the 4 th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random)? 3?? 4 1 Lowest sample? Actual Minimum

54 Actual Maximum Highest sample 2 Q. What is the chance of the 4 th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random) 25% chance higher than highest seen 3 25% lower than highest and higher than second highest 25% higher than lowest and lower than second lowest 4 Actual Minimum 1 Lowest sample 25% lower than lowest seen A. 50% % = (1 - (1 / n 1)) * 100

55 Actual Maximum Q. What is the chance of the 12 th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random) Highest sample % chance higher than highest seen? Actual Minimum Lowest sample 5% lower than lowest seen A. 90%? % = (1 - (1 / n 1)) * 100

56 # Prior Samples Prediction Next Sample Within Prior Sample Range 3 50% 4 67% 5 75% 6 80% 7 83% 8 86% 9 88% 10 89% 11 90% 12 91% 13 92% 15 93% 17 94% 20 95%

57 What distribution fits cycle time data and why THE SHAPE OF CYCLE TIME 62 Commercial in confidence

58 Probability Density Function.32 Note: Histogram from actual data x Histogram Gamma (3P) Lognormal Rayleigh Weibull 63 Commercial in confidence

59 Probability Density Function x Histogram Weibull 64 Commercial in confidence

60 Normal, or it will be after a few thousand more simulations

61 Base + 1 Delay

62 Base + 2 Delays

63 Base + 3 Delays

64 Base + 4 Delays

65 Base + 5 Delays

66 ASSESSING AND COMMUNICATING RISK 72 Commercial in confidence

67 Speaking Risk To Executives Buy them a copy of Flaw of Averages Show them you are tracking & managing risk Do We are 95% certain of hitting date x With 1 week of analysis, that may drop to date y We identified risk x, y & z that we will track weekly Don t Give them a date without likelihood February 29 th 2013 Give them a date without risk factors considered To do the backlog of features, February 29 th, 2013

68 **Major risk events have the predominate role in deciding where deliver actually occurs ** We spend all our time estimating here Plan Performance Issues External Vendor Delay 1 2 3

69 Risk likelihood changes constantly 95 th Confidence Interval 1 2 3

70 Risk likelihood changes constantly 95 th Confidence Interval 1 2 3

71 Risk likelihood changes constantly 95 th Confidence Interval 1 2 3

72 Risk likelihood changes constantly 95 th Confidence Interval 1 2 3

73 Models Start with Assumptions Cycle Time Guessed Last releases performance Another teams performance Amount of Work # Stories guessed Scope creep guessed Similar sizing distribution to last release assumed Concurrent WIP Guessed Measured from last release Similar process to last release assumed 80 Commercial in confidence

74 Cost to Develop Staff A : $$$$$$$$ Staff B : $$ Actual Date A Planned / Due Date Actual Date B Actual Date C Staff C : $ Staff Option A Staff Option B Staff Option C 81 July August September October November December Forecast Completion Date Commercial in confidence

75 Can t I just use Average Cycle Time? Forecast (Days) Median 525 Mean 670 MC >50% 677 MC >85% 793 MC >95% 851 Projection of cycle time to complete 25 stories in a sequential fashion based upon historical cycle time data 82 Commercial in confidence

76 Total Cycle Time Story / Feature Inception 5 Days Waiting in Backlog 25 days Active Development 30 days Waiting for Release Window 5 Days System Regression Testing & Staging 5 Days

77 Test Dev Lead Time Story / Feature Inception 5 Days Waiting in Backlog 25 days Active Development 30 days Waiting for Release Window 5 Days System Regression Testing & Staging 5 Days

78 1. Historical Cycle Time Design Develop Test A Process to Combine Multiple Uncertain Measurements / Estimates is Needed Design Develop Test 2. Planned Resources/ Effort 4. Historical Scope Creep Rate (optional) The Work (Backlog) Backlog Feature 1 Feature 2 Feature 3 Commercial in confidence 5. Historical Defect Rate & Cycle Times (optional)