Cycle Time Forecasting. Fast #NoEstimate Forecasting
|
|
- Elizabeth Davidson
- 5 years ago
- Views:
Transcription
1 Cycle Time Forecasting Fast #NoEstimate Forecasting
2 15 th August 2013 Forecasts are attempts to answer questions about future events $1,234,000 Staff 2 Commercial in confidence
3 We estimated every task for project x and have determined it will take days Oh, over 3 months, so we ll do Project Y instead Perfect information is rarely sought or needed 3 Commercial in confidence
4 Estimate or #NoEstimate? Prioritized by Opinion / Mandatory Team Exists & Fixed in size? Date Fixed or New Team? No Estimate! 70% Chance Forecast Team Size/Skillset Options you need to Economic Prioritization Used? Forecast Date Fixed or Calculated New Team? (You just Value performed / Cost of a probabilistic forecast! Delay? 3.5 cases out of 5) Team Fixed in size? Foggy Value / Cost of Delay? Foggy Estimate GUESS! Forecast Team Size/Skillset Options Forecast Date, Manage Risks
5 Economic Prioritization same time, different value Value eg. Cost of Not Doing Time Rem. Build Time/Effort Complete Order? Product 1 3 Product 2 2 Product 3 1
6 Economic Prioritization same value different time Value eg. Cost of Not Doing Time Rem. Build Time/Effort Complete Order? Product 1 1 Product 2 2 Product 3 3
7 Suggested Economic Prioritization Cost of Delay W.S.J.F. = Prioritization Heuristic to optimize cashflow Do Highest First Duration or Time Delayed (often, the remaining time to complete) Learn this here You are here to learn this
8 8 Commercial in confidence
9 There is NO single forecast result Uncertainty In = Uncertainty Out There will always be many possible results, some more likely and this is the key to proper forecasting
10 Likelihood Probabilistic Forecasting combines many uncertain inputs to find many possible outcomes, and what outcomes are more likely than others 50% Possible Outcomes 50% Possible Outcomes Time to Complete Backlog 10 Commercial in confidence
11 Likelihood Probabilistic Forecasting combines many uncertain inputs to find many possible outcomes, and what outcomes are more likely than others 85% Possible Outcomes 15% Time to Complete Backlog 11 Commercial in confidence
12 How certain based on model forecast Further calculations to make economic tradeoffs 12 Commercial in confidence
13 Staff Dev Cost Cost of Delay Decision Induced Uncertainty Planned / Due Date Actual Date 13 July August September October November December Forecast Completion Date Commercial in confidence
14 How I Present Cycle Time Forecasts When will my feature ship? Current - 1 Devs Description Team 5 Jul 29 Jul Feature 1 25 Jul 16 Aug Feature 2 3 Aug 9 Sep Feature 3 9 Sep 4 Oct Feature 4 2 Oct 2 Nov Feature 5 10 Nov 7 Jan Feature 6 8 Jan 8 Mar Feature 7 14 What are my staffing options? Project Staffing Options Full time staff Current team Dev Staff Test Staff All Features Complete > 85% CI Mar Jan Tester Dec Devs Oct Tester Oct-2013
15 What is modelling and how to use cycle time MODELING AND CYCLE TIME 15 Use with attribution
16 A model is a tool used to mimic a real world process Models are tools for low-cost experimentation
17 Success Hinges on Being Wrong Early Initial Goal To enable decisions of staff size, cost and date for prioritization Accuracy: Eventual duration within the forecast set Next Goal To mimic the real system accurately enough to perform reliable experiments (sensitivity testing) To give the earliest warning of actual versus plan diff Early action have bigger impact Accuracy: Getting better all the time (matching actual) 17 Commercial in confidence
18 Diagnostic Simple Depth of Forecasting models Linear Projection System Cycle Time Partitioned Cycle Time Simulated process 18 Commercial in confidence
19 Simple Cycle Time Model Amount of Work (# stories) Cycle Time Random Chance / Risk 19 Commercial in confidence Parallel Work in Proc. (WIP)
20 Cycle Time = Date Complete Date Started If you have a captured start and complete date you can calculate cycle time. Note: Cycle time sometimes referred to as Lead Time but not going to discuss this now! 20 Use with attribution
21 6. Phases 1. Historical Cycle Time 2. Planned Resources/ Effort Monte Carlo Analysis = Process to Combine Multiple Uncertain Measurements / Estimates 4. Historical Scope Creep Rate (optional) The Work (Backlog) Backlog Feature 1 Feature 2 Feature 3 Commercial in confidence 5. Historical Defect Rate and Cycle Times (optional)
22 Historical Story Lead Time Trend Sum Random Numbers More often Less often Sum: Basic Cycle Time Forecast Monte Carlo Process 1. Gather historical story lead-times 2. Build a set of random numbers based on pattern 3. Sum a random number for each remaining story to build a potential outcome 4. Repeat many times to find the likelihood (odds) to build a pattern of likelihood outcomes Total Days = Sum ( Story n Random n ) Effort Days To Complete
23 23 Commercial in confidence
24 24 Commercial in confidence
25 FORECASTING STRATEGIES 25 Commercial in confidence
26 When you have historical data 1. Model Baseline using historically known truths 2. Test Model against historically known truths 3. Forecast The Past The Future
27 Compare Model vs Actual Often Number of Completed Features (Model vs Actual) Start Month 1 Month 2 Month 3 Month 4 Month 5 Model Actual
28 Forecast Trend Report Card Watch GREAT Watch Ok Bad Watch WORRY Watch Cycle Time Trend versus Model ACTIVE WIP Actual vs Model Story Count Trend versus Model Arrow Legend: Green: Heading in beneficial direction Red: Heading in damaging direction For Watch the WIP and story count trends need to be analyzed. They are 1:1 ratio of compensating importance, and may offset each other. 28 Commercial in confidence
29 When you have no historical data The Future
30 Simple Cycle Time Model Cycle Time (range guess or samples) Amount of Work (# stories, scope creep, defects) Parallel Work in Proc. (WIP on board) 31 Commercial in confidence
31 If we understand how cycle time is statistically distributed, then an initial guess of maximum allows an inference to be made Alternatives - Borrow a similar project s data Borrow industry data Fake it until you make it (AKA guess range) 33 Use with attribution
32 Probability Density Function.32 Note: Histogram from actual data x Histogram Gamma (3P) Lognormal Rayleigh Weibull 34 Commercial in confidence
33 Exponential Distribution (Weibull shape = 1) The person who gets the work can complete the work Teams with no external dependencies Teams doing repetitive work E.g. DevOps, Database teams, 35 Commercial in confidence
34 Weibull Distribution (shape = 1.5) Typical dev team ranges between 1.2 and Commercial in confidence
35 Rayleigh Distribution (Weibull shape = 2) Teams with MANY external dependencies Teams that have many delays and re-work. E.g. Test teams 37 Commercial in confidence
36 Probability Density Function Shape How Fat the distribution. 1.5 is a good starting point f(x) Location The Lower Bound Scale How Wide in Range. Related to the Upper Bound. *Rough* Guess: (High Low) / x Histogram Weibull 38 Commercial in confidence
37 What Distribution To Use... No Data at All, or Less than < 11 Samples (why 11?) Uniform Range with Boundaries Guessed (safest) Weibull Range with Boundaries Guessed (likely) 11 to 30 Samples Uniform Range with Boundaries at 5 th and 95 th CI Weibull Range with Boundaries at 5 th and 95 th CI More than 30 Samples Use historical data as bootstrap reference Curve Fitting software 39 Commercial in confidence
38 Questions Download the slides (soon) and software at Contact me Read: 40
39 Why traditional forecasting practices fail to deliver and why this is the only outcome possible CURRENT FORECASTING PRACTICES 41 Commercial in confidence
40 Development Time 5 Days (~ 15%) Total Story Lead Time 30 days Blocked and Waiting Time 9 Days Testing Time 2 Days Waiting Time 3 Days Defect Rework 2 Days Waiting Time 8 Days Release / DevOps Time 1 Day
41 Development Time 5 Days (~ 15%) Total Story Lead Time 30 days Blocked and Waiting Time 9 Days Testing Time 2 Days Waiting Time 3 Days Defect Rework 2 Days Waiting Time 8 Days Release / DevOps Time 1 Day
42 Pre Work 30 days Story / Feature Inception 5 Days Waiting in Backlog 25 days Total Story Lead Time Active Development 30 days 30 days Post Work 10 days Waiting for Release Window 5 Days System Regression Testing & Staging 5 Days
43 Pre Work 30 days Story / Feature Inception 5 Days Waiting in Backlog 25 days Total Story Lead Time 30 days Post Work 10 days 9 days (70 total) approx 13% Waiting for Release Window 5 Days System Regression Testing & Staging 5 Days Active Development 30 days
44 How to use historic data to estimate the different distributions ESTIMATING DISTRIBUTIONS USING HISTORIC DATA 48 Commercial in confidence
45 What Distribution To Use... No Data at All, or Less than < 11 Samples (why 11?) Uniform Range with Boundaries Guessed (safest) Weibull Range with Boundaries Guessed (likely) 11 to 50 Samples Uniform Range with Boundaries at 5 th and 95 th CI Weibull Range with Boundaries at 5 th and 95 th CI Bootstapping (Random Sampling with Replacement) More than 100 Samples Use historical data at random without replacement Curve Fitting 49 Commercial in confidence
46 Tools EasyFit from $499 R Free Statistics feature of KanbanSim Free Excel of course Monte Carlo features of KanbanSim $ Google for Monte Carlo Simulation Tools Oracle, Palisade, HubbardResearch (he may be here, so a big recommendation!) 50 Commercial in confidence
47 Sampling at Random Strategies If you pick what samples to use, you bias the prediction Strategies for proper random sampling Use something you know is random (dice, darts) Pick two groups using your chosen technique and compute your prediction separately and compare Don t pre-filter to remove outliers Don t sort the data, in fact randomize more if possible 51 Commercial in confidence
48 Estimating Concurrent Effort from Cumulative Flow Chart Concurrent WIP Sample : Find the smallest and the biggest or take at least 11 samples to be 90% sure of range 52 Commercial in confidence
49 20% 40% 60% 80%
50 Scope Creep Over Time Look at the rate new scope is added over time 55 Commercial in confidence
51 What is the likelihood the next sample I take will be within the range of previous samples PREDICTION INTERVALS
52 Actual Maximum Q. What is the chance of the 4 th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random) Actual Minimum
53 Actual Maximum Highest sample 2 Q. What is the chance of the 4 th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random)? 3?? 4 1 Lowest sample? Actual Minimum
54 Actual Maximum Highest sample 2 Q. What is the chance of the 4 th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random) 25% chance higher than highest seen 3 25% lower than highest and higher than second highest 25% higher than lowest and lower than second lowest 4 Actual Minimum 1 Lowest sample 25% lower than lowest seen A. 50% % = (1 - (1 / n 1)) * 100
55 Actual Maximum Q. What is the chance of the 12 th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random) Highest sample % chance higher than highest seen? Actual Minimum Lowest sample 5% lower than lowest seen A. 90%? % = (1 - (1 / n 1)) * 100
56 # Prior Samples Prediction Next Sample Within Prior Sample Range 3 50% 4 67% 5 75% 6 80% 7 83% 8 86% 9 88% 10 89% 11 90% 12 91% 13 92% 15 93% 17 94% 20 95%
57 What distribution fits cycle time data and why THE SHAPE OF CYCLE TIME 62 Commercial in confidence
58 Probability Density Function.32 Note: Histogram from actual data x Histogram Gamma (3P) Lognormal Rayleigh Weibull 63 Commercial in confidence
59 Probability Density Function x Histogram Weibull 64 Commercial in confidence
60 Normal, or it will be after a few thousand more simulations
61 Base + 1 Delay
62 Base + 2 Delays
63 Base + 3 Delays
64 Base + 4 Delays
65 Base + 5 Delays
66 ASSESSING AND COMMUNICATING RISK 72 Commercial in confidence
67 Speaking Risk To Executives Buy them a copy of Flaw of Averages Show them you are tracking & managing risk Do We are 95% certain of hitting date x With 1 week of analysis, that may drop to date y We identified risk x, y & z that we will track weekly Don t Give them a date without likelihood February 29 th 2013 Give them a date without risk factors considered To do the backlog of features, February 29 th, 2013
68 **Major risk events have the predominate role in deciding where deliver actually occurs ** We spend all our time estimating here Plan Performance Issues External Vendor Delay 1 2 3
69 Risk likelihood changes constantly 95 th Confidence Interval 1 2 3
70 Risk likelihood changes constantly 95 th Confidence Interval 1 2 3
71 Risk likelihood changes constantly 95 th Confidence Interval 1 2 3
72 Risk likelihood changes constantly 95 th Confidence Interval 1 2 3
73 Models Start with Assumptions Cycle Time Guessed Last releases performance Another teams performance Amount of Work # Stories guessed Scope creep guessed Similar sizing distribution to last release assumed Concurrent WIP Guessed Measured from last release Similar process to last release assumed 80 Commercial in confidence
74 Cost to Develop Staff A : $$$$$$$$ Staff B : $$ Actual Date A Planned / Due Date Actual Date B Actual Date C Staff C : $ Staff Option A Staff Option B Staff Option C 81 July August September October November December Forecast Completion Date Commercial in confidence
75 Can t I just use Average Cycle Time? Forecast (Days) Median 525 Mean 670 MC >50% 677 MC >85% 793 MC >95% 851 Projection of cycle time to complete 25 stories in a sequential fashion based upon historical cycle time data 82 Commercial in confidence
76 Total Cycle Time Story / Feature Inception 5 Days Waiting in Backlog 25 days Active Development 30 days Waiting for Release Window 5 Days System Regression Testing & Staging 5 Days
77 Test Dev Lead Time Story / Feature Inception 5 Days Waiting in Backlog 25 days Active Development 30 days Waiting for Release Window 5 Days System Regression Testing & Staging 5 Days
78 1. Historical Cycle Time Design Develop Test A Process to Combine Multiple Uncertain Measurements / Estimates is Needed Design Develop Test 2. Planned Resources/ Effort 4. Historical Scope Creep Rate (optional) The Work (Backlog) Backlog Feature 1 Feature 2 Feature 3 Commercial in confidence 5. Historical Defect Rate & Cycle Times (optional)