Impact of Consumption Demand Data on DLA Forecast Accuracy

Transcription

1 Impact of Consumption Demand Data on DLA Forecast Accuracy Presented By Vivek Kumar Robert Lo Linda Tsang Project Advisor: Robin Roundy Project Sponsor: Eric Gentsch 1

2 Agenda Background Study Objectives Measures of Accuracy Data Generic Approach Double Exponential Smoothing Model Fourier Model Results Conclusions & Recommendations 2

3 Background End-user demand = Consumption Demand DLA demand = Wholesale Demand Currently, DLA uses only historical Wholesale Demand to forecast The Supply Chain Suppliers DLA Military Services Warehouses Maintenance Centers 3

4 Agenda Background Study Objectives Measures of Accuracy Data Generic Approach Double Exponential Smoothing Model Fourier Model Results Conclusions & Recommendations 4

5 Study Objectives Primary Objective Evaluate the impact of having consumption demand data on DLA s forecast accuracy of wholesale demand Secondary Objective Analyze the impact of transitioning from using Double Exponential Smoothing to Fourier forecasting 5

6 Agenda Background Study Objectives Measures of Accuracy Data Generic Approach Double Exponential Smoothing Model Fourier Model Results Conclusions & Recommendations 6

7 Measures of Accuracy 1. Mean Absolute Deviation (MAD) MAD = 2. Mean Squared Error (MSE) MSE = 3. Mean Absolute Percentage Error (MAPE) MAPE = 1 T e t T t= 1 1 T T t= 1 t= 2 e t T 1 et T 1 D t T - total number of time periods D t - Demand in period t F t - Forecast in period t e t = F t D t error in period t 7

8 Measures of Accuracy Metric Features for a Single Part Features for Aggregated Parts MAD MSE MAPE High demand time periods usually carry more weight. More weight is given to the large errors. High and low demand time periods have equal weight. High demand parts are more important than low demand parts. Large errors have much more influence than small errors. High and low demand parts will have equal influence. 8

9 Agenda Background Study Objectives Measures of Accuracy Data Generic Approach Double Exponential Smoothing Model Fourier Model Results Conclusions & Recommendations 9

10 Data DLA items with only Air Force demand Wholesale and consumption demand data Wholesale data from DLA SAMMS Consumption data from USAF D035K and SBSS Given transactional data and converted into monthly and quarterly data Given 16,000 wholesale replenishment parts Given consumption demand data for most parts Examine items with both wholesale and consumption demand data (16,000 13,000) 10

11 Data Unit cost data: Mean: $ Std. Deviation: $ Median: $9.98 Many low-cost parts Dollarize demand data More expensive parts have more weight Select random sample of 2,000 based on their average annual dollarized demand 11

12 Data Wholesale Demand Pareto Chart for All Parts Pareto Chart for 2000 Parts Cumulative % Dollar Demand % parts contribute to 93% of demand Cumulative %Number of Parts Cumulative % Dollar Demand % parts contribute to 93% of demand Cumulative %Number of Parts The distributions of the two populations are very similar. 12

13 Data Consumption Demand Pareto Chart for All Parts Pareto Chart for 2000 Parts Cumulative % $-demand % parts contribute to 95% of demand Cumulative % Number of Parts Cumulative % $-demand % parts contribute to 95% of demand Cumulative % Number of Parts The distributions of the two populations are very similar. 13

14 Data Coefficient of Variation - Standard deviation / mean COVs for All Parts COVs for Selected 2000 Parts y c n e u q e r F y c n 150 e u q e r 100 F Coefficient of Variation Coefficient of Variation 4.50 Mean: 1.27 Standard Deviation: 0.81 Median: 1.05 Mean: 1.26 Standard Deviation: 0.83 Median: 1.01 The COVs for the 2000 parts are comparable to those for all parts. 14

15 Data Lead Time Time between when an order is placed and when it is received Lead Times for All Parts Lead Times for Selected 2000 Parts y c n e u q e r F y c n 400 e u q e 300 r F Lead Time (month) Lead Time (month) Mean: 6.89 months Standard deviation: 3.76 Median: 6 months Mean: 6.42 months Standard deviation: 3.23 Median: 7 months The 2000 parts are representative of the population. 15

16 Agenda Background Study Objectives Measures of Accuracy Data Generic Approach Double Exponential Smoothing Model Fourier Model Results Conclusions & Recommendations 16

17 Generic Approach Overall objective: Determine whether incorporating consumption data will improve DLA s forecast accuracy over a part s lead time Lead Time Demand The Demand over the part s Lead Time Lead Time Forecast The Forecast generated in time t for time t+τ, where τ is the Lead Time Rationale of examining lead time accuracy Drives Safety Stock investments 17

18 Generic Approach Objective function Maximize forecast accuracy for all parts over each part s lead-time by: Minimize Lead-time MAD Minimize Lead-time MSE Minimize Lead-time MAPE Robust minimization How do we incorporate consumption demand? 1) Weight 2) Weighted Offset 18

19 Generic Approach Weight Method D t = w * WS t + (1-w) * C t D t = Combined Demand Stream WS t = Wholesale Demand in time t C t = Consumption Demand in time t w = Weight to give to Wholesale Demand w Є [0,1] 19

20 Generic Approach Weighted Offset Method Rationale: Consumption Demand may occur first Buffering effect of Military Service Warehouse Better representation of true demand for a part Model for demand: D t = w * W t + (1-w) * C t-x x is the number of months consumption data is offset w is the weight as described earlier offset of x = 1 to 5 months 20

21 Generic Approach Weight Method (offset 0) Offset 0 Time periods away from present Wholesale Demand Consumption Demand Time periods away from present 21

22 Generic Approach Weighted Offset Method Offset 1 Wholesale Demand Consumption Demand Offset 2 Wholesale Demand Consumption Demand Time periods away from present 22

23 Generic Approach Calculating the objective (Forecast Accuracy) Total of 6 years (72 months) of demand data Use first 36 months (12 quarters) to initialize forecasts Update forecast every period Depends on property of model Number of forecasts generated depends on how often the forecasts are updated 23

24 Agenda Background Study Objectives Measures of Accuracy Data Generic Approach Double Exponential Smoothing Model Fourier Model Results Conclusions & Recommendations 24

25 Double Exponential Smoothing Model (DES) Overcomes the limitations of moving averages Useful when the historical data series is not stationary, but contains a trend Used for adjusting a forecast to reflect the error in the current forecast Parameter (α) determines how much weight to place on the current observation of demand and on past observations 25

26 Double Exponential Smoothing Model Algorithm: 1. Compute the single smoothed quantity S t St = αxt + ( 1 α) St 1 where x t = demand in time t [2] 2. Compute the double smoothed quantity S t [2] [2] S αs + 1 α) S t = t ( t 1 3. Forecast for τ periods into the future F γ γ t [2] t+ τ = ( 2 + ) St (1 + ) St where α γ = τ 1 α 26

27 Double Exponential Smoothing Model Calculating the objective (Forecast Accuracy) 12 quarters of data Forecast for Lead Time Demand Quarters away from present 0 13 quarters of data Forecast for Lead Time Demand Quarters away from present 27

28 Double Exponential Smoothing Model 11 different lead time forecasts generate 11 different forecast errors Average the 11 absolute values of the errors MAD for a part MSE, MAPE also calculated accordingly Average Metric over all parts Aggregate Lead-Time Accuracy Measure = The Objective 28

29 Agenda Background Study Objectives Measures of Accuracy Data Generic Approach Double Exponential Smoothing Model Fourier Model Results Conclusions & Recommendations 29

30 Fourier Forecasting Model Transition from legacy to Fourier Forecasting Main purpose of Fourier: Capture Seasonality Basic idea behind Fourier Forecasting: Use of Fourier Series Fourier Series: Sinusoidal curve that is a collection of sine/cosine functions of varying frequencies Demand with Forecasts Original demand data Our results Demand Time 30

31 Fourier Forecasting Model Our algorithm in laymen s terms Step 1) Input a certain amount of historical data points into the model Demand Data Demand Time 31

32 Fourier Forecasting Model Step 2) Remove any trend using simple linear regression Demand Dem and Data Time Demand Data trend line detrended data 32

33 Fourier Forecasting Model Step 3) Fit a Fourier series to the data via a Fourier Transform Fourier Transform: An algorithm that converts time data into frequency data Demand Demand Data Time detrended data Sine curve with frequency 0.1 and amplitude Frequency Domain 6 Amplitude Sine curve with frequency 0.4 and amplitude Fre que ncy 33

34 Fourier Forecasting Model Amplitud Amplitud Frequency Domain Frequency Frequency Domain (filtered) Frequency Our specific Fourier Model: Low-pass filter Filters out curves from series with certain frequencies to create a smooth demand stream that follows the data and accounts for seasonality Filter out frequencies above 0.05 cycles per month Property of Low-pass filter Note: Simple version of Manugistics implementation 34

35 Fourier Forecasting Model Another Fourier Model: Spectral Filter Amplitud Frequency Domain Frequency Filters out curves from series that have an amplitude below a certain level For example: 3 Frequency Domain Amplitud Frequency 35

36 Fourier Forecasting Model Demand Demand with Forecasts Time Original demand data Fourier Curve Perform an inverse Fourier Transform on the data to convert the Fourier series back into meaningful demand data. Add back the trend that was removed in step 2 Demand with Forecasts Original demand data Our results Demand Time 36

37 Fourier Forecasting Model Extend the series a desired amount into the future. (forecasts) Demand with Forecasts Original demand data Our results Demand Time 37

38 Fourier Forecasting Model Calculating the objective (Forecast Accuracy) 36 months of data Forecast for Lead Time Demand Months away from present 0 36 months of data Forecast for Lead Time Demand Months away from present 0 38

39 Fourier Forecasting Model different lead time forecasts generate different forecast errors Average the absolute values of the errors MAD for a part MSE, MAPE calculated accordingly Average Metric over all parts Aggregate Lead-Time Accuracy Measure = The Objective 39

40 Agenda Background Study Objectives Measures of Accuracy Data Generic Approach Double Exponential Smoothing Model Fourier Model Results Conclusions & Recommendations 40

41 Value of Consumption Data DES Baseline DES Baseline Parameter α=0.2 Using Quarterly Wholesale Demand only Average lead time for each part to be 2 quarters DES Baseline Results MAD: $ 10,122 MSE: $ 2 19,310,298,979 MAPE:

42 Value of Consumption Data-DES Pareto Charts show that most parts are low dollar-demand parts MAD and MSE dominated by high dollar-demand parts MAPE dominated by low dollar-demand parts We can adjust two variables Weight Offset We examine three different scenarios compared to the baseline Consumption only Weight Weighted offset 42

43 Value of Consumption Demand Data DES Results Minimize MAD Minimize MSE % Improvement from baseline 60% 50% 40% 30% 20% 10% 0% -10% -20% 9.78% 5.88% % Consump Only Weight Weighted Offset % Improvement from baseline 60% 50% 40% 30% 20% 10% 0% 47.12% 16.85% 21.16% Consump Only Weight Weighted Offset Minimize MAPE Best Configurations % Improvement from baseline 60% 50% 40% 30% 20% 10% 0% 25.57% 25.57% 25.69% Consump Only Weight Weighted Offset Metric MAD MSE MAPE Weight Offset (months)

44 DES Observations Master of Engineering Project MAD DES Results Offset 0 Offset 1 Offset 2 Offset 3 Offset 4 Offset 5 The best weights for all offsets are approximately Weight on Wholesale Demand MAPE DES Results Offset 0 Offset 1 Offset 2 Offset 3 Offset 4 Offset 5 Low dollar-demand parts dominate MAPE MAPE strongly influenced by Consumption demand Weight on Wholesale Demand 44

45 DES Results Summary Consumption data are useful Both offsets and weights are important! MAD improves from baseline by 10% MSE improves from baseline by 47% MAPE improves from baseline by 25% Low dollar-demand parts are different from high dollar-demand parts Consumption data is better for low dollar-demand parts 45

46 Value of Consumption Demand Data DES Results Offset by 4 months, using 50% Wholesale and 50% Consumption % Improvement from baseline 55.00% 45.00% 35.00% 25.00% 15.00% 5.00% -5.00% % 3.25% 47.12% 10.94% DES Robust MAD Optimal MSE Optimal MAPE Optimal % MAD MSE MAPE Robust results give a configuration that performs well over all three measures of accuracy 46

47 Fourier Baseline Fourier Baseline Main difference: monthly demand data Baseline forecast: wholesale data only Fourier Baseline Results MAD: $ 14,010 MSE: $ 2 10,254,642,836 MAPE:

48 Value of Consumption Demand Data Fourier Results Minimize MAD Minimize MSE Improvement from baseline 60% 50% 40% 30% 20% 10% 0% -10% -20% 11.7% 10.5% -7.4% Consump Only Weight Weighted Offset Improvement from baseline 60% 50% 40% 30% 20% 10% 0% 16.9% 7.4% 10.6% Consump Only Weight Weighted Offset Minimize MAPE Optimal Configurations Improvement from baseline 60% 50% 40% 30% 20% 10% 0% 35.4% 35.4% 34.8% Consump Only Weight Weighted Offset Metric MAD MSE MAPE Weight Offset (months)

49 Fourier Observations Master of Engineering Project MAD Fourier R esults `` Offset 0 Offset 1 Offset 2 Offset 3 Offset 4 Offset 5 MAD for the best weight of each offset all within 2% of each other and all 13% better than baseline Weight On Wholesale D emand MAPE Fourier Results Offset 0 Offset 1 Offset 2 Offset 3 Offset 4 Offset 5 Time delay for low dollar-demand parts is short MAPE strongly influenced by Consumption demand Weight on Wholesale Demand 49

50 Fourier Results Summary Consumption data are useful Both offsets and weights are important! MAD improves from baseline by 11.70% MSE improves from baseline by 16.90% MAPE improves from baseline by 35.43% Low dollar-demand parts are different from high dollar-demand parts Consumption data is better for low dollar-demand parts 50

51 Value of Consumption Demand Data Fourier Results Offset by 5 months, using 50% Wholesale and 50% Consumption 60.00% % Improvement from baseline 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% 10.92% 16.90% 24.49% Fourier Robust MAD Optimal MSE Optimal MAPE Optimal % MAD MSE MAPE All optimal methods use some form of consumption data 51

52 Rough Model Comparisons DES Robust Results MAD: $ 9,793 MSE: $ 2 10,212,036,401 MAPE: 1.50 Vs. Fourier Robust Results MAD: $ 12,480 MSE: $ 2 8,522,108,998 MAPE: 1.58 Potential Reasons for Difference 1 month vs. 1 qtr updates. Monthly vs. Quarterly Demand Data Nature of Exponential Smoothing Over-fitting? We think not. Seasonality is real Our Fourier Model is a simplified version of Manugistics Fourier Model Note : MSE, MAPE more important for Manugistics 52

53 DES Exploratory Experiments Experiment: Transformation Ordinary DES Historical Demand D t DES Forecast F t Forecast Error F t - D t MAD DES with Power Transformation 0<p 1 Transformation d t = (D t ) p Transformation F t = (f t ) (1/p) Historical Demand D t Compressed Forecast f t Forecast Error F t - D t MAD 53

54 DES Exploratory Experiments Experiment: Transformation Exploratory Experiment 3 Results Overall, optimal p = 1, which means transformation does not improve forecast accuracy Impact of transformation on individual cluster: COV 0-1: Optimal p = 1, no transformation needed COV 1-2: Optimal p = 1, no transformation needed COV 2-3: Optimal p = 1, no transformation needed COV 3-4: Optimal p = 1, no transformation needed COV 4-5: Optimal p = 1/5, transformation improves forecast accuracy (MAD) from baseline by 80.78% Transformation on only high COV parts improves forecast accuracy (measured by MAD) 54

55 Agenda Background Study Objectives Measures of Accuracy Data Generic Approach Double Exponential Smoothing Model Fourier Model Results Conclusions & Recommendations 55

56 Conclusions The use of consumption data improves forecast accuracy Conclusions 56 R obust c onf i g ura t i on pe rf orms we ll a c ross a ll t hre e me a sure s of a c c ura c y Depending on metric, different configurations are optimal

57 Recommendations Metric MAD MSE MAPE DES % Improvement from baseline 9.78 % % % Fourier % Improvement from baseline % % % DES Robust Improvement MAD: 3.25% MSE: 47.12% MAPE: 10.94% Fourier Robust Improvement MAD: 10.92% MSE: 16.9% MAPE: 24.49% 57

58 Recommendations A one-size-fits-all approach may not be appropriate Reason: MAPE is very different from the MAD and MSE in the usage of consumption demand High demand parts should be treated differently than low demand parts Should look into segmentation of parts by COV and volume and apply different ways to forecast (transformation) 58

59 Implementation Issues Lowpass filter vs. Manugistics Scalability to millions of parts [Lack of] consumption data for all items, all customers 59

60 Future Project Suggestions Implement Manugistics Fourier forecasting method Evaluate the value of improved forecast accuracy on both supply chain and inventory performance 60

61 We would like to thank the DLA Aging Systems Program for sponsoring this project. 61

62 Double Exponential Smoothing Model Initialization of Algorithm: [2] Compute initial values for and using the T periods of available data S S 0 S 0 1 α [2] 1 α = aˆ(0) ˆ(0) S0 = aˆ(0) 2 bˆ(0 ) α 0 b where â (0) and ˆb(0 ) can be estimated from the intercept and slope of the regression line of the available data. α 62

63 DES Exploratory Experiments Experiment 2: Clustering By COV Objective: To see if using different weight for each segment of parts will improve forecast accuracy (MAD) Exploratory Experiment 2 Results Weight Method with clustering s MAD: $9, Improvement from Weight method without clustering s MAD by only 1.9% Experiment 3: Finding Optimal DES parameter α by minimizing MAD Objective: To see if DLA is using the optimal parameter Exploratory Experiment 1 Results Optimal parameter = 0.247, very close to DLA s 0.2 MAD: $9,878.55, improves baseline MAD by 2.41% MSE: $18,602,791,842, improves baseline MSE by 3.66% MAPE: 1.62, improves baseline MAPE by 4.14% The computation time it takes to cluster by COV may not be worth the 1.9% improvement 63