There has been a lot of interest lately

Transcription

1 BENCHMARKING SALES FORECASTING PERFORMANCE MEASURES By Kenneth B. Kahn Although different companies use different measures to evaluate the performance of forecasts, mean absolute % error (MAPE) is the most popular one by and large, error is the lowest on an industry level, and the highest, on a location level satisfied companies are more likely to measure forecast accuracy than others. There has been a lot of interest lately in determining benchmarks for sales forecast accuracy. However, present benchmarks are mostly aggregated across industries and do not distinguish between the ways for calculating accuracy. Consequently, present benchmarks do not offer clear benchmarks for accuracy on an industry and/or performance measurement basis. In an attempt to clarify such benchmarks, a study was undertaken by Georgia Tech s Marketing Analysis Laboratory. Like previous studies, the study intended to determine what methods companies are using to calculate forecast accuracy, what degrees of forecast accuracy are being achieved, how forecast accuracy is being reported, and what factors appear to drive forecast accuracy. However, unlike previous studies which have mostly aggregated industry and forecast measurement statistics into one set of benchmark criteria, the present study considered sales forecast accuracy on a market-specific basis, e.g., consumer product companies, and forecast performance measurement basis, e.g., accuracy achieved using mean absolute percent error. METHODOLOGY Data for the study were collected using a mail survey with follow-up telephone interviews. Eighty sales forecasting managers were surveyed. Following two survey waves, 40 managers replied and were subsequently interviewed (50% response rate). Interviews conducted with responding managers, clarified responses, and asked additional open-ended questions. Responding firms were primarily consumer product firms (75% of KENNETH B. KAHN Dr. Kahn is an Assistant Professor of Marketing in Georgia Institute of Technology s Dupree School of Management. He has widely published. His research interests include product planning, product management, marketing analysis, and sales forecasting. He currently serves as the Director of Georgia Tech s Marketing Analysis Laboratory. respondents), the remaining firms (25%) primarily served industrial markets. Respondents indicated a high degree of competition, moderate degree of technological change, and high degree of promotions being used. Respondents also indicated that their products were madeto-forecast and that the average number of forecasts made per period was 4,877 forecasts to support on average 62 distribution outlets. The average length of delivery cycle was four weeks, with a nineweek average raw material lead time, and an eight week average production lead time. The average product shelf life was 100 weeks. RESULTS Analyses comprised descriptive statistics (average, standard deviation, minimum value, maximum value), correlations, and/or crosstabulation. Below are results for the four given research questions: How is forecast accuracy measured? What level of forecast accuracy is being achieved? How is forecast accuracy reported? And what factors appear to impact forecast accuracy? How Is Forecast Accuracy Measured? Forty percent of respondents measure sales forecast error, twenty-five percent measure accuracy, twenty percent measure both accuracy and error, and fifteen percent do not measure sales forecast performance at all. Of those that measure accuracy or error, mean absolute percent error is the most popular statistic, followed by mean absolute deviation (MAD) and standard deviation. Overwhelmingly, these respondents indicated that their sales forecast accuracy/error statistic was weighted by THE JOURNAL OF BUSINESS FORECASTING, WINTER

2 volume. (See Tables 1 and 2) Note that in accuracy measurement, forecast accuracy is expressed as a percentage; for example, the forecast is 90% accurate. Inventory statistics included line fill and order-fill statistics. Interestingly, half of the companies using MAPE employed a formula that used forecast as the denominator,whiletheother half of companies employed the textbook version of the formula, which prescribes the use of actual as the denominator. (See Table 3) It should be noted that each of these two formulas can introduce a bias into the performance measurement process. In particular, the use of the forecast as the denominator encourages overforecasting because a larger forecast reduces reported forecast error(overforecasting corresponds to higher inventory levels). Conversely, the use of actual encourages underforecasting because a higher actual reduces reported forecast error (underforecasting corresponds to higher stockout situations). Thus, one should seriously consider the implications of each MAPE formula and choose the one that best fits company history and policies. What Level of Forecast Accuracy Is Being Achieved? Forecast accuracy was reported at five forecast levels. These five levels are defined as follows: 1. Industry Level Forecasts are forecasts of the overall industry/marketplace. 2. Corporate Level Forecasts are forecasts of company activity when all products and services are aggregated together. 3. Strategic Business Unit Level Forecasts are forecasts of individual business unit activity within the company. 4. Stock Keeping Units (SKU) Level Forecasts are forecasts of individual product lines or product items. 5. Location Level (SKU/Location) Forecasts are forecasts of distribution center/ customer level activity per product line/ item. Not accounting for type of market or type of forecast performance measure, the average level of accuracy across industry, TABLE 1 SALES FORECASTING PERFORMANCE MEASURE Performance Measure Number of Responses % Responses Accuracy Error Both Accuracy and Error 8 20 Accuracy/Error Not Measured 6 15 TABLE 2 MEASURES OF FORECAST ACCURACY/ERROR USED Metric Used Number of Responses % Responses Mean Absolute Percent Error (MAPE) Mean Absolute Deviation (MAD) 6 13 Mean Squared Error (MSE) 1 2 Inventory Statistics 2 4 Standard Deviation 6 13 Other TABLE 3 FORMULA FOR THE ACCURACY/ERROR STATISTIC EMPLOYED Formula Number of Responses % Responses MAPE with Actual as Denominator MAPE with Forecast as Denominator Both Types of MAPE Calculations Used 2 5 Ratio of Forecast to Actual 1 3 Absolute Difference Between Actual and Forecast 1 3 Don t Know How Error Statistic Calculated 1 3 Forecast Accuracy/Error Not Measured 6 15 Not Given 5 12 corporate, strategic business unit (SBU), stock keeping unit (SKU), and location levels was 95%, 91%, 86%, 77%, and 70%, respectively. Companies using MAPE reflected 90%, 96%, 88%, 77%, and 67% accuracy across industry, corporate, strategic business unit (SBU), stock keeping unit (SKU), and location levels, respectively. Those not using MAPE reflected 95%, 83%, 79%, 83%, and 98%, respectively (due to the low sample sizes, no definitive conclusions can be made with regards to differences between MAPE and non-mape users). In consumer product firms forecast accuracy across industry, corporate, SBU, SKU, and location levels was 90%, 92%, 90%, 76%, and 67%, respectively. Consumer product firms using MAPE reflected forecast accuracy across industry, corporate, SBU, SKU, and locations levels as 90%, 96%, 90%, 77%, and 67%, respectively; too few a number of consumer product firms not using MAPE reported sales forecast accuracy, which precludes a valid calculation of accuracy across levels for this group. Most confidence in the above benchmarks can be placed on forecast accuracy at the SKU level because of the larger number of responses at this forecast level.(see Table 4) 20 THE JOURNAL OF BUSINESS FORECASTING, WINTER

3 TABLE 4 SALES FORECASTING PERFORMANCE ACROSS FORECAST LEVELS Forecast Level Overall % Forecast % Forecast % Forecast % Forecast % Forecast Accuracy Accuracy Accuracy Accuracy Accuracy Achieved By Achieved Achieved Achieved By Achieved Companies By Companies By CPC CPC Using Using MAPE Not Using MAPE MAPE Industry Level sd = 4.75 sd = n/a sd =n/a sd =n/a sd = n/a n = 3 n = 1 n =1 n =1 n = 1 Corporate Level sd = sd = 2.78 sd =20.03 sd =11.66 sd = 2.90 n = 14 n = 9 n =3 n =10 n = 8 Strategic Business Unit Level sd = sd = sd =19.01 sd =3.82 sd = 3.82 n = 13 n = 10 n =3 n =8 n = 8 Stock Keeping Unit Level sd = 9.92 sd = 9.70 sd =14.19 sd =9.18 sd = 9.63 n = 24 n = 19 n =3 n =20 n = 17 Location Level sd = sd = 9.13 sd =n/a sd =9.73 sd = 9.63 n = 10 n = 9 n =1 n =8 n = 8 Notes: CPC = Consumer product companies sd = Standard Deviation. n = Number of Responses. A comparison of accuracy by consumer product firms using the two different ways of calculating MAPE was undertaken to determine if choice of formula actually bias performance measurement. Unfortunately, the small sample size inhibited a valid comparison of the two groups, thus results are inconclusive. Further work is needed to clarify this issue. (See Table 5) How Is Forecast Accuracy/Error Reported? Companies indicated that they used both statistical output and graphs to report sales forecasting results. Although multiple responses were allowed, the majority of responses indicated that such reporting is mostly at the national SKU and/or national SBU level. The overwhelming majority of reporting occurred on a monthly basis. Most companies indicated that most, if not all, departments were given free access to both reports and graphs. In most cases, the sales forecasting department had the responsibility to develop and disseminate such reports and/or graphs. What Factors Appear to Impact Forecast Accuracy? Correlation analyses were applied, but did not suggest any significant factors that could possibly drive forecast accuracy. The only significant correlation was degree of competition and sales forecast accuracy less competition, higher accuracy. Obviously, this does not lend itself as a prescription for improving forecast accuracy/error. The lack of common findings across companies, aside from competition, suggests that sales forecast performance is impacted by companyspecific and market-specific forces. Generic guidelines for sales forecast accuracy/error improvement therefore cannot be determined at this time. Further analysis is needed in this area. Satisfied Versus Dissatisfied Sales Forecasting Companies Findings indicated 38% of respondents (15 companies) were dissatisfied and 45% of respondents (17 companies) were satisfied; the remaining 17% (8 companies) of respondents were neutral. THE JOURNAL OF BUSINESS FORECASTING, WINTER

4 Due to the number of dissatisfied versus satisfied companies, there was interest in determining whether performance measurement factors stimulate satisfaction. Additionally, there was interest in determining what practices are common to satisfied versus dissatisfied sales forecasting companies, thereby serving as a surrogate to best(most satisfying) practices. Surprisingly, only two companies were very satisfied with their process. Through interviews, it was found that the majority of satisfied companies use a consensus approach in their sales forecasting process. Of the 17 satisfied companies, nine had one department develop a baseline forecast which is then updated in a consensus meeting with other departments. Two of the satisfied companies answered that each department develops its own forecast and a committee develops the final compromise forecast. This contrasts the 15 dissatisfied companies. Twelve of these companies stated that they did not use a consensus approach to sales forecasting: six of these twelve companies had each department develop and use their own separate forecasts, the other six had one department develop a single forecast that all departments used. Satisfied companies also were more likely to recognize a separate sales forecasting department responsible for the forecasting function. Furthermore, they were more likely to hold the forecasting department accountable for forecasting accuracy (76% of satisfied companies versus 27% of dissatisfied companies). In addition, 88 % of satisfied companies indicated that the sales forecasting department has high credibility, only 33% of dissatisfied firms indicated that the sales forecasting department has high credibility. Most impressively, satisfied companies were more likely to measure forecast accuracy, error, or both. Of the 17 satisfied companies, 94% measure accuracy, error or both: 24% measure accuracy, 35% measure error, and 35% TABLE 5 SALES FORECASTING PERFORMANCE BASED ON DIFFERENT MAPE FORMULAE USED Forecast Level % Forecast % Forecast Accuracy for Accuracy for CPC Using CPC Using MAPE Formula MAPE Formula Based on Actual Based on Forecast Industry Level sd =n/a n =1 n =0 Corporate Level sd =3.74 sd =1.16 n =5 n =3 Strategic Business Unit Level sd =4.08 sd =3.82 n = n =3 Stock Keeping Unit Level sd =8.58 sd =10.76 n = n =10 Location Level sd =6.97 sd =19.45 n =6 n =2 measure both. Only 6% of satisfied companies did not measure accuracy or error. In contrast, only 60% of dissatisfied companies measure accuracy or error, with 20% measuring accuracy and 40% measuring error. The remaining 40% of dissatisfied companies indicated that they do not measure forecast accuracy or error at all. MAPE was the predominant performance measurement statistic for satisfied companies. Half of the satisfied companies used actual in the denominator of the MAPE equation, and the other half used forecast in the denominator. Interestingly, the two very satisfied companies indicated that both ways to calculate MAPE are employed. Interviews with these companies indicated that their interest was to establish a range of forecast performance, versus establish a specific performance number. CONCLUSIONS Results from this benchmarking study of forty companies indicates several benchmarks for sales forecasting performance measurement. They are as follows: Mean absolute percent error (MAPE) is the most popular sales forecast accuracy measure. Most companies weight their sales forecast accuracy/error statistic by volume. Forecast performance measurement is typically reported on a monthly basis at a national SKU level. Forecast accuracy at the stock keeping unit level on average across all companies is 77%. Consumer product companies reflect an accuracy level of 76% at this level. Consumer product companies using MAPE reflect an accuracy of 75% at this level. Forecast accuracy appears to be driven 22 THE JOURNAL OF BUSINESS FORECASTING, WINTER

5 by company-specific and marketspecific factors(primarily competition), with the latter mostly beyond the control of companies. Companies satisfied with their sales forecasting process are distinguished from those companies that are dissatisfied in regards to a variety of characteristics. Using satisfaction as a surrogate to better performance, and possibly, an initial step towards better performance, the following is recommended based on the below characteristics which were reflected by satisfied companies: Measure forecast accuracy or error. Use a consensus approach in your sales forecasting process. Recognize the sales forecasting department as a separate entity, responsible for all forecasting functions. Hold the forecasting department accountable for forecasting accuracy. Ensure that the forecasting department has high credibility. Overall, study findings highlight that it is imperative to implement some type of performance measurement. Failure to measure any sales forecast performance instills dissatisfaction with the forecasting process, and more importantly, inhibits companies ability to react quickly and effectively to market forces. The present study, while limited in size, offers important considerations for the implementation of new initiatives or enhancement of current practices related to sales forecasting performance measurement. (The author gratefully acknowledges the financial support of Amgen, Apple South, Borden, Mary Kay, Nabisco, and Polaroid for conducting this study.) TEN COMMANDMENTS... (Continued from page 2) remember one monthly forecast meeting I attended. In that meeting, the production person said that now we could manufacture and deliver customized steel doors in three weeks instead of four weeks. Salesperson immediately responded that in that case we could get more orders. Because he was not accepting orders where delivery was required in three weeks. By circulating preliminary forecasts, forecaster can obtain the reaction of the forecast users. If the forecaster does not agree with them, he or she will know how to defend at the time of final presentation. The forecasting manager of Fujitsu America finds this procedure very helpful. Most of the differences are resolved before the final presentation. The forecaster can further improve the involvement of the forecast users if he or she periodically follows up how forecasts are used, and if not, why? This will help not only to bring the forecaster and customers closer but also enable the forecaster to determine exactly kinds of forecasts they need. EDUCATE THE FORECAST USERS It is important that the forecast users understand the basics of business forecasting. The more they know, the more they will appreciate. What they need to know is the basic concepts, not algorithm, how forecasts are prepared. What assumption is made when a time series model is used, that is, the past pattern will continue? What assumptions are made when a cause and effect model is used? Assumptions might have been made about the advertising budget, price, competitive action, and state of the economy. The forecast users have to understand that forecasts are must for making a business plan. The option is either to use your own forecasts which may be highly crude or use the ones prepared by the professional. They have to understand that forecasts are neither a goal nor a plan, but based on a plan. They can to some extent influence the future by changing the plan. They have to understand that forecasts are not entirely mechanical, a certain amount of judgment goes into them. The more the forecast users understand the forecasting, the more they will appreciate and use them. The forecaster can play an important part in educating them. The education of the forecast users can achieved by offering a seminar on forecasting and/or explaining something about it at the organization s regularly scheduled meeting. Carroll Mohan of Coca- Cola says that one way to accomplish this is to find a rock in an organization, a person who is likely to be around for a while, with and through whom professional understanding can be developed. No matter which approach you use, bear in mind, managers want to learn but they don t want to be taught. PROVIDE DETAILED REPORT ON FORECASTS The buy-in of forecasts depends very much on, among other things, how forecasts are reported and presented. Here are some cardinal rules of reporting forecasts. i. Give forecasts in as much detail as needed. ii. Indicate the assumptions used in preparing forecasts. iii. If forecasts deviate substantially from iv. the norm, give reasons. In some situations, forecasts under different scenarios may be needed. v. Make sure forecasts are internally consistent, that is, each column adds up correctly. Forecast users often have a built in suspicion about forecasts. If one thing is wrong, then the whole thing will be considered wrong. vi. Give forecasts along with actuals. This way they can see where we are and where we are going. vii. Standardized the format of forecasting report. This way they will know right away where they can find what they are looking for. PRESENT FORECASTS IN A PROFESSIONAL MANNER How forecasts are presented can make the difference. Good presentation can turn the skeptics into believer. Here are some cardinal rules of presenting forecasts: i. Describe forecasts in simple, jargonfree language. Each business has its own culture, terminology and language. Use their terms and language in making presentation. ii. Don t prove, demonstrate. A rigorous, deductive proof carries immense persuasion to another yogi but not to a kommissar. Show how good your forecasts were in the past by displaying graphically both actual and projected THE JOURNAL OF BUSINESS FORECASTING, WINTER