Forecasting Revenues in Ancillary Markets

Transcription

1 Forecasting Revenues in Ancillary Markets Ajay Patel and Eddie Solares

2 Forecasting Revenues in Ancillary Markets 1 Summary Most companies use an expected value formula to forecast revenues from a sales or project opportunity pipeline. The expected value is typically calculated as the aggregate of the estimated likelihood of the opportunity being real P(go), times the likelihood of the company winning the opportunity P(win), times the estimated annual project revenue. This approach is accurate when two conditions are met: 1) when there are a large number of opportunities in the pipeline and 2) when timing of the project is well known. However, when companies start to rely on new or ancillary markets for a larger share of the future revenues, these two conditions of the traditional forecasting approach are rarely met for a variety of reasons. Though new and ancillary markets may be a significant source of future revenues, companies typically focus on few, large projects to optimize their investment (thus, any single outcome can significantly alter the results from an expected value), and may not have the necessary insight to realistically estimate the factors in the expected value formula, especially timing. This leads to the statistics behind the traditional expected value formula breaking down and giving erroneous results. This could lead to dramatic surprises in financial results from a win or loss of a single large project that was incorporated in the expected value based forecast. Thus, traditional forecasting techniques are not adequate nor well suited to predict future revenues for firms pursuing growth in new and ancillary markets. In this study, SMA s analysis shows that the traditional expected value approach systematically over-estimates forecasts by as much as 60% in the near-term, and creates more than a 58% likelihood of a significant negative revenue surprise. The study shows that project timing is the largest source of forecast inaccuracy for a typical project pipeline. For the illustrative pipeline example used in the study, the small sample size of high value projects typical of pursuing an ancillary market accounted for approximately 12.5% of the forecast inaccuracy, whereas uncertainty in project timing is responsible for 50% (the remainder is from the inherent uncertainty modeled by P(go) and P(win)). The study proposes a method to incorporate project timing uncertainty as a fourth factor P(t) in the expected value formula. The proposed approach can be easily implemented by the finance department, and as a component to monitoring business development activities. If it were the case that a project decision and contract award was always as planned, then within one standard deviation we would expect that the results of the traditional expected value formula would be equal to the actual results that occur in real life. When you add the uncertainty that the contract might not be awarded the year it is scheduled, a clearer picture emerges as to why the traditional approach is not as robust as we would expect it to be. Using a Monte-Carlo method to simulate real life scenarios, the study shows that by considering realistic delays we over-estimate near-term revenues by 20% to 60% using the traditional expected value formula. The more disconcerting result is that within one standard deviation, the forecast can be as good as 10% or as bad as 150%. These are not the kind of results the finance division of a company wants to be dealing with when planning company revenues. So if timing is an issue how can we address it without completely scrapping the traditional approach? In the study, we develop a unique approach to the traditional expected value approach where we introduce a matrix that incorporates a timing delay on a year-by-year basis. The matrix is designed to only take in one additional timing probability to simplify the approach. Running the same Monte-Carlo simulation with this added matrix, we get a new expected value formula that improves accuracy by as much as 57% and reduces the likelihood of any surprise by 30% within one standard deviation. This added fix to the traditional approach can be modified and tailored to suit historical data and fit multiple types of timing delays. It also is a relatively easy fix to the traditional approach and does not overly complicate an already uncertain process that has been known to be difficult to pin down.

3 Forecasting Revenues in Ancillary Markets 2 Contents 1. Introduction How Forecasts are Traditionally Calculated The Challenge of Ancillary Markets Study Design Overview Static Pipeline Design for International Opportunities in A&D Creating Realistic Futures (i.e. Simulations) Timing: The Hidden Third Parameter Obstacles in Forecasting The New Expected Value Formula Expected Value vs. New Expected Value Analysis Conclusions and Recommendations Appendix A Appendix B About the Authors Ajay Patel is President and CEO of SMA with over 30 years of strategy consulting, business development, operations, program management, and systems engineering experience. He holds an MBA in Strategic Planning and Finance from USC and a BS Physics from John Hopkins University. Eddie Solares is a management consulting analyst at SMA with research experience focused on statistical analysis of large datasets using programming languages. He holds a MS in Physics from UCLA and a BS in Astrophysics from UCSC.

4 Forecasting Revenues in Ancillary Markets 3 1. Introduction Most companies today rely on revenues from ancillary or adjacent markets as a part of their core growth strategy. Revenues from these pursuits have been typically difficult to forecast, largely because of misunderstood nuances of those markets especially customer buying behaviors and processes. To a new entrant, and even established competitors, these markets can appear to have murky decision mechanisms, informal influence networks, and significant uncertainties with customer budgets and needs. This has been particularly true for project-driven industries pursuing international opportunities, such as Aerospace and Defense, Engineering and Construction, and public sector consultancies. These pursuits drive critical decisions on allocation of resources, especially for large projects that the company is relying on to make their numbers. Our experience working with clients across many industry sectors has been that revenue forecasting in these markets has been persistently challenging, even for companies that have been in ancillary markets for many years. The factors that are considered include estimates of the value and scope of the opportunity, the uncertainty of the project (or purchase) moving forward, the firm s competitive position and timing of the award. Traditional approaches to revenue forecasting clearly do not work for these types of markets. The most widely used approach is an expected value calculation for each opportunity that discounts the annual revenue estimate by a probability of the project moving forward and a probability of winning the contract. The reason why this traditional approach fails in ancillary markets is that the opportunities typically tend to be large in revenue and few in number, thus the uncertainties and outcome of any individual opportunity can easily affect the overall revenue forecast and financial result. To compensate for these uncertainties and the possibility of a surprise from a single project outcome (win or loss), experienced firms typically use a rule-based or ad hoc decision as to whether to include or not include a specific opportunity in the forecast. Though this approach compensates for overall uncertainty, it also creates a systemic bias and relies heavily on individual judgement resulting in a process where the forecasting accuracy is not predictable. See Appendix A for more details. In this paper, we demonstrate through statistical simulations that the traditional forecasting approach does not accurately predict realistic scenarios. The traditional approach systematically over-estimates forecasts by 22 54% in the near-term, and creates more than a 58% likelihood of a significant negative revenue surprise. We address the shortcomings of the traditional approach with an easily implemented new expected value formula that improves accuracy by as much as 57% and reduces the likelihood of a surprise by 30%. The new formula accounts for the following attributes: 1) the probability of the project being real, 2) the probability of being awarded the project and 3) the probability of the contract being awarded at the planned start year. We use an example of an aerospace and defense firm pursuing international projects for illustration; but the approach can be implemented to any ancillary markets or industry sector. Note that this paper applies only to revenue forecasting from new opportunities, and does not address revenues from existing backlog.

5 Forecasting Revenues in Ancillary Markets 4 2. How Forecasts are Traditionally Calculated The traditional approach to revenue forecasting is to use the expected value formula. The expected value is an anticipated value for a given sales opportunity. The general expected value (EV) formula in revenue forecasting is given by EV j [t] = P j R j [t] (1) where P is the probability of opportunity j, and R is revenue of the opportunity at some time t. Unfortunately, if either the opportunity itself is not certain or the confidence of winning is wrong, then the forecast can be significantly misrepresented. For this reason, most companies uncouple these two factors into individual probabilities: the probability of the opportunity occurring and the probability that the opportunity is won. In this manner, the EV formula then becomes a realistic revenue forecasting method. The modified formula uses these two probability variables as a compounded probability: P j = P j (go) P j (win go = 1) The probability P j (win go = 1) is the conditional probability of winning given that the opportunity is real (i.e. the project or purchase actually proceeds; whereas P j (go) is the independent probability of the project or purchase occurring. This enables an intuitive approach to estimating the confidence of each opportunity in the revenue forecast. Taking all the opportunities in a pipeline, the sum of all the EV revenues gives the total forecasted revenue. Therefore, the total revenue forecast F from an opportunity j (of a total of n opportunities in the sales pipeline) for a given year i can be written as n n F i = j=1 EV j [t i ] = j=1 P j (go) P j (win go = 1) R j [t i ] (2) where EV is the expected value, t is time, P are the probabilities and R is the revenue for the opportunity. The traditional expected value method is a simple calculation, only requires two probability estimates, and is intuitive for management and finance departments. However, it is a statistical approach and requires a large number of opportunities for the statistics to give accurate results which stems from the mathematical law of large numbers.

6 Forecasting Revenues in Ancillary Markets 5 3. The Challenge of Ancillary Markets When pursuing sales opportunities in ancillary markets, there are sources of uncertainty that are hard to manage. This is in comparison to core markets where the company has years of experience understanding buying processes and has experiential knowledge that can be applied to make reasonable judgements on the viability of a project and the firms competitive position to win, namely P(go) and P(win go = 1). When pursing and competing in ancillary markets, as in Figure 1 for example, lack of selling experience, lack of deep customer intimacy and lack of access to influence networks make it difficult to ascertain project viability and competitive position. Figure 1: Classification scheme tailored to revenue planning for new market opportunities. Although category E is included for completeness, it is excluded in study once opportunities are classified. Category B Revenue 4-5 Year Revenue Meaningful financial impact starting on the fourth year Medium Outlook Good P(go) probability and decent P(win) probability Good Outlook High P(go) probability and good P(win) probability High Outlook High P(go) and P(win), most likely won opportunity Category A Revenue Category C Revenue Category D Revenue Category E Dropped No financial impact within the first five years

7 Forecasting Revenues in Ancillary Markets 6 Furthermore, some markets have inherent uncertainties with regard to customer needs, budgets, project timing, and other factors that further complicate an already complex situation. This is particularly true when pursuing international public-sector projects. As an example, aerospace and defense firms have been pursing growth in international markets for the past five years as a key complement to their core domestic business. These are typically large projects, but few in number where winning or losing a handful of contracts can have an appreciable impact on revenues. Companies in many industry sectors have faced similar obstacles on their path to globalization: inability to make progress within the formal or imputed buying process, blind spots that arise from pre-existing biases, influence levers that are obscured or concealed, and general difficulty discerning signal from noise as an outsider looking in. As these firms rely more on international sales, these obstacles make forecasting revenue with reasonable certainty difficult. The probabilities P(go) and P(win), the program value and the start year of these opportunities, are much less certain for international opportunities. On top of that, they are relatively few in number but each with a much larger potential value creating the potential for significant revenue surprises from a single loss or win. Therefore, in order to create a realistic pipeline we suggest that companies include these factors explicitly as a way to categorize and prioritize opportunities in their sales pipeline. An example screening process of a firm pursing international opportunities as an ancillary market is illustrated in Figure 1. In this example, five categories to group each individual opportunity are created to tackle the problem. The categories defined are generically referred to A, B, C, D, and E, each has distinct attributes of timing, program viability and competitive position. We use this structure to model traditional and alternate forecasting methodologies. Our models are limited to a five-year horizon, so opportunities in category E are dropped from this study due to its lack of impact in the revenue planning horizon.

8 Forecasting Revenues in Ancillary Markets 7 4. Study Design Overview Our study systematically tests the traditional approaches of forecasting revenue at a detailed level to gain insight into the sources of error and develop improved techniques. For our example in Figure 1 of penetrating foreign markets, the following is an outline of our study design: 1. A static sample pipeline is created as a baseline to test the accuracy of different forecasting techniques. The pipeline consists of 1,052 opportunities (1,000 domestic-market and 52 international-market opportunities). This sample size choice was to simulate the small number of opportunities in an ancillary market, compared to the large number of core-market opportunities. For each opportunity, the P(go) and P(win) probabilities are assigned according to pipeline category shown in Figure The accuracy of the traditional forecasting method is tested by creating 1,000 future scenarios of the pipeline using a Monte-Carlo technique. Each future scenario is a statistical simulation of each opportunity going forward as a real program or purchase, and whether the company wins or loses the competition for the opportunity. These binary outcomes are tested against the expected value calculation. The difference between the sum of the expected values and the sum of the binary revenues is a measure of the accuracy of the forecasting technique. 3. We analyze the sources of variability to better understand the accuracy of the traditional forecasting technique. We demonstrate that most significant source of forecasting errors (and systemic revenue misses) is with the estimate of the program or purchase planned start year versus actual start year. 4. Finally, an alternative forecasting method is developed to improve forecasting accuracy by considering a third probability parameter that takes into account the predicted program start year versus the actual start year of the program.

9 Forecasting Revenues in Ancillary Markets 8 5. Static Pipeline Design for International Opportunities in A&D The first step in our outline in Section 4 requires us to design the pipeline for our international market example. A pipeline of new business opportunities is created in order to test the different forecasting method approaches. A typical pipeline for a public sector contractor that is relying on international opportunities as part of their core revenues is emulated. To create a methodical way to categorize new market opportunities, a classification scheme based on five categories that allows the differentiation and assignment of realistic variables for the probabilities P(go) and P(win) is created. As stated Section 4, 1000 domestic opportunities (250 in each category A/B/C/D) and 52 international opportunities (13 in each category A/B/C/D) are created. We randomly assign a revenue value of $100 million, $500 million, or $750 million for all 1,000 domestic opportunities giving a total domestic revenue of $212,500 million. Similarly, revenue values of $500 million, $750 million, or $1,500 million for each of the 52 international opportunities is assigned at random totaling $41,750 million in international revenues. In this model, the international pipeline is 16% of the total. The distribution of these revenues can be seen in Figure 2. These values are kept static throughout the study. Figure 2: Revenue distribution across 1,000 domestic and 52 international opportunities. Opportunity Value, $ million Domestic Number of Opportunities Total Revenue, $ million Opportunity Value, $ million International Number of Opportunities Total Revenue, $ million $ $75,000 $ $12,500 $ $100,000 $ $11,250 $ $37,500 $1, $18,000 Totals 1000 $212,500 Totals 52 $41,750 From Figure 2, we can also see that the international opportunities are fewer quantity and larger in revenue value. In general, the number of opportunities in ancillary markets is usually much less than core markets. This comes with the territory of exploring and penetrating new markets. The larger revenue reflects how companies prioritize opportunities; they generally seek new opportunities where the revenue is worth the risk of entering ancillary markets. Now that the revenue distribution is established, start years for each opportunity are assigned in order to configure our pipeline. As Figure 3 shows, the 5-year pipeline is simulated with each opportunity having a different start date. The opportunities in categories A, B, and C are apportioned to start in different years based on a random seed such that 70% of the opportunities will start in 2018, 20% will start in 2019 and 10% start in Figure 3: Start year percentage distribution based on category for each opportunity. Category Start Year Percentage Distribution A/B/C 70% 20% 10% 0% 0% D 0% 0% 0% 70% 30% For category D, based on our definition of the category in Figure 1, it is seen that this opportunity does not have a meaningful impact in the first three years so the start year percentages do not begin until the year % of the opportunities in category D are assigned to start in 2021 and 30% start in Note that category E is excluded from the study since by design those opportunities have no significant

10 Revenue ($ million) Forecasting Revenues in Ancillary Markets 9 revenue impact in the first five years. Once start years are assigned to each opportunity, for both domestic and international, they are kept static throughout the course of the study. The last step to finalize the baseline pipeline is to distribute the recognition of revenue for each opportunity over the years that the program is executed. We assume that each program is executed over a 5-- year period and has a simple increasing profile defined in Figure 4. The revenue recognized in year 1 is 2 /20th of the total, increasing to 6 /20th in year 5, resulting in recognizing 100% of the revenue over 5 years. The increasing program revenue profile is not necessarily typical across all types of programs or purchases, but compensates for the burn-off of backlog resulting in an overall growth forecast for the firm. Figure 4: Revenue distribution for compensation of burn-off backlog of program opportunities. Year Year 1 Year 2 Year 3 Year 4 Year 5 Revenue Distribution 2 / 20 x Revenue 3 / 20 x Revenue 4 / 20 x Revenue 5 / 20 x Revenue 6 / 20 x Revenue For visualization purposes, Figure 5 shows a graph of the revenue distribution by category for domestic and international. Note that the total international revenue is about a quarter of the total domestic revenue. Figure 5: Total revenue for both domestic and international category. $250,000 $200,000 Domestic and International A B C D $150,000 $100,000 $50,000 $0 Domestic International By design, an even distribution split among categories is noted. Recall that of the 1,000 domestic opportunities, 250 were assigned to each category and for the 52 international opportunities, 13 were assigned to each category. To view the ramp up in revenue the graph of the revenue distribution by domestic and international we graph it in Figure 6 and by the categories in Figure 7. Note, that in alignment to the model design, category D does not begin until after 2020.

11 Revenue ($MM) Forecasting Revenues in Ancillary Markets 10 Figure 6: Revenue distribution per year for domestic and international opportunities. $90,000 $80,000 $70,000 Domestic International Revenue ($ million) $60,000 $50,000 $40,000 $30,000 $20,000 $10,000 $ Figure 7: Revenue distribution per year based on category distribution. $90,000 $80,000 $70,000 A B C D $60,000 $50,000 $40,000 $30,000 $20,000 $10,000 $ Now that the category, revenue, and start year distributed across all five years of our planning horizon are created, probabilities are assigned for each opportunity. A systematic way of assigning the probability of the opportunity occurring P(go) and the probability of winning the opportunity P(win) needs to be created. Recall that in Figure 1, categories with qualitative definitions were classified among opportunities. In order to quantify the P(go) and P(win), a percentage range is assigned to each category for both probabilities. The model must also take into account that ancillary markets have much smaller probabilities due to the inherent risks a company takes in doing business in unknown markets.

12 Forecasting Revenues in Ancillary Markets 11 To assign probabilities based on our category distribution, probability ranges are assigned according to Figure 8. These probabilities are connected to the attributes of each pipeline category as defined in Figure 1. A random sampling based on these ranges is done and assigned a probability within the range. Consistent with how the parameters are established, the probabilities of P(go) and P(win) are kept static throughout the duration of the study. Figure 8: Probability ranges for each category based on type of opportunity and category of opportunity. Category E opportunities are removed from study but included for completeness. Domestic International Category Probability (Go) Probability (Win) A 90% 100% 80% 90% B 80% 100% 70% 80% C 70% 100% 50% 70% D 50% 100% 50% 70% E < 50% < 50% A 70% 100% 80% 90% B 60% 100% 70% 80% C 50% 100% 50% 70% D 30% 100% 50% 70% E < 30% < 50% Figure 9 column headings reflect all static variables that have been created so far in our pipeline. The expected value is calculated in Figure 9 using the formula EV = P(go) P(win) Revenue. This is done for all 1,000 domestic opportunities and all 52 international opportunities. These values are all kept static for the remainder of the simulation in order to keep the relative comparisons free of any statistical bias. Note that the expected value is less than the total pipeline, which makes sense since not all opportunities will result in revenue. The reader is reminded that the aggregate sum of all expected values of the opportunities should equal the realistic revenue stream given a large enough sample size. Figure 10 graphs the expected revenue from all opportunities over our 5-year planning horizon. As described above, this is the aggregate of the expected revenue from each opportunity estimated by multiplying the recognized revenue in each year by the two probabilities. Now that the static pipeline is fixed, statistical simulations can be conducted and results tested to see whether or not the expected value is an accurate measure for forecasting revenue. Forecasted revenue will be used interchangeably with the expected value (EV). Figure 9: Sample of static pipeline created showing opportunity number, whether domestic D or international I, category, planned start year, two probabilities assigned to them, total opportunity revenue and expected value (EV) calculated from traditional formula. Opportunity Dom/Int Category Year P(go) P(win) Revenue $ million EV, $ million 1001 D A % 83.52% $100 $ D A % 82.45% $500 $ D A % 88.53% $500 $ D A % 87.78% $500 $ D A % 83.73% $100 $ D A % 81.50% $500 $ D A % 84.01% $100 $ D A % 88.56% $100 $88.42

13 Forecasting Revenues in Ancillary Markets 12 Figure 10: Sum of EV throughout five years for both international and domestic. $50,000 $45,000 $40,000 Domestic International Revenue ($ million) $35,000 $30,000 $25,000 $20,000 $15,000 $10,000 $5,000 $

14 Forecasting Revenues in Ancillary Markets Creating Realistic Futures (i.e. Simulations) Moving on to Step 2 of the study approach (see Section 4), testing of the traditional forecasting methodology is done to see how well the expectation value formula predicts future revenues. Future scenarios are created by sampling from the probability distributions defined in the previous section; each scenario represents the collection of outcomes of all opportunities determined from the sampling. For each opportunity, a random number is generated representing the outcome for the variable P(go) to determine the outcome pertinent to whether the project will proceed or not. If the random number is equal to or is less than the defined P(go), then the project will be awarded (i.e. it is assigned a value of 1) and if is greater than P(go) then the project was not real (i.e. it is assigned a value of 0). We do the same for P(win). To avoid any statistical correlations, different random samples for each probability are used. As an example, if a random sample of s = 0.78 is generated and compared to a probability of P(go) = 0.92, it is seen see that 0.78 < 0.92 and thus falls within the probability, so a binary value of B(go) = 1 is assigned. If, however, a random sample of s = 0.96 is generated then it is greater that P(go) = 0.92 and thus assigned the binary value of B(go) = 0. The actual scenario value (SV) is then calculated by SV = B(go) B(win) Revenue. This is done for all 1,000 domestic opportunities and all 52 opportunities. From Figure 11, it is seen that if either or both opportunities get a binary value of 0, the entire scenario value is $0.00. This scenario value simulates an actual real-life scenario where the opportunity is either won or lost. Figure 11: Pipeline with random sample drawn which assigns binary 0 or 1 to B(go) and B(win) to calculate Scenario Value (SV). Opportunity Dom/Int Category Year P(go) P(win) B(go) B(win) Scenario Value, $ million 1245 D A % 89.63% 1 1 $ D A % 87.38% 1 1 $ D A % 86.47% 1 1 $ D A % 88.52% 1 1 $ D A % 80.41% 1 1 $ D A % 84.69% 1 0 $ D B % 74.05% 1 1 $ D B % 74.91% 0 1 $ D B % 77.04% 1 1 $ D B % 79.65% 1 0 $ D B % 70.23% 1 0 $ D B % 77.34% 1 1 $100 From the law of large numbers, given a large enough sample size, it is expected that the sum of all the expected values (EV) equals the sum of all scenario values (SV), that is SV EV = 0. The sum of all of the EV and the sum of all SV in our pipeline is taken and the difference between them for each of the five years give us our percent difference. That is, the percent difference can be written as % Diff = Σ(SV EV) = [B(go) B(win) - P(go) P(win)] R (3) where R is the revenue. This is done for both the domestic and the international opportunities. This process counts as one simulation that produces one value for the percent difference. In order to generate a standard deviation and determine the accuracy of our results 1,000 simulations are done.

15 Forecasting Revenues in Ancillary Markets 14 Once the 1,000 runs are completed, the average of these is taken along with the corresponding standard deviation of each. The mean value is calculated and an error bar of one standard deviation is assigned for each of the five. The process for running one of these simulations can be quite long so the Python programming language is used to quickly run 1,000 of these simulations. 1 Running and plotting the percent differences between the expected value (EV) and the scenario value (SV) for all 1,000 runs and taking the average percent difference gives us the results of Figure 12. Forecast for core domestic markets is spot on and there is very little mean error between the differences. This means that SV = EV on average and there is very little variation within the error bars. The standard deviation, or error bar, is less than 5% for the domestic market and is relatively small which reflects the high number of opportunities. Figure 12: Percent difference between the expected value (EV) and the scenario value (SV) with one standard deviation as the error bar Domestic International Combined 0.15 Percent Difference (1 - EV/S) From a mathematical standpoint, this reflects the law of large numbers since there is a large sample size of 1,000 opportunities the results tend to not deviate as much since the EV approaches SV as the sample size nears infinity. The mean percent difference for the international ancillary market is also relatively small. It is still slightly larger than for the domestic opportunities, but the difference is within 5% points of the 0% difference. However, the error bar is considerably larger than the domestic one with values greater than 15%. This is reflective of the sample size since there are fewer opportunities, 52 to be exact, and each individual opportunity has a more significant impact on the revenue. A point should be made to state that the size of the error bar is what indicates whether the traditional approach has flaws namely, if any single future scenario could result in a significant revenue forecast miss. This is mostly 1 Programming languages are useful to do routine calculations in a quick and efficient way. Since the simulation is run 1,000 times, it is much easier for a computer to do this job as opposed to manually recalculating the results one by one. High-level programming languages are useful for these types of calculations as they are designed for general purpose programming. The programming language used in this study is Python, but there are other highlevel languages such as C++ and Java that can similarly be used.

16 Forecasting Revenues in Ancillary Markets 15 driven by the wide range of possible outcomes for smaller number but larger in value of the international opportunities. However, from these results two things can be incorrectly assumed: 1. The traditional expected value represented in Equation 1 is a good way to forecast revenue; 2. Having a larger sample size reduces the standard deviation of your results. We will next show that the first assumption is incorrect since there is a critical hidden parameter that is assumed to be true that is not considered. We will also address the second assumption in our recommendations for an improved forecasting approach. It will be seen that the true error bars of the traditional forecasting method defined by Equation 1 will be large and the flaws of these results will be evident once the hidden parameter is introduced.

17 Forecasting Revenues in Ancillary Markets Timing: The Hidden Third Parameter The results for the traditional expected value seem acceptable notwithstanding the possible wide range of outcomes, but is missing an important aspect of forecasting. The timing of the opportunity is critical to forecasting revenues and is the additional hidden parameter that failed to be taken into account. This source of uncertainty will now be considered in the next step of the analysis (see Section 4). In the prior analysis, it was assumed that the planned start year was the actual start year of the opportunity. From experience, this is not realistic especially in public sector driven markets and in ancillary markets where there is less budgetary and decision transparency. It is also known that forecasters have a significant systemic bias for optimism. There is a tendency for companies to over-estimate their revenue in forecasting models for new markets and timing plays an important factor. Thus, it is crucial to estimate the probability for the planned start year to be the actual start year. This introduces a new statistical parameter when running the future scenarios that was not considered before. The question now becomes how is uncertainty in timing simulated? when running the future scenarios. For each opportunity, when the dice is rolled for each future scenario value, not only is the probability of whether or not the program happens and if it is won simulated but also when it occurs. To do this a probability is associated with each start year and the remaining probabilities are pushed out to later years. That is, there is a certain probability it starts in the planned year and if it does not fall under this probability the project gets pushed out a year or two down the line. Using the timing probability matrix in Figure 13, probabilities are assigned for each of the five years of forecasting revenues. From the matrix, it is seen that for domestic opportunities with a planned year of 2021 there is a 65% chance it actually starts in 2021, a 20% chance that it starts in 2022, and a 15% chance that it occurs in 2023 or later. Since only five years revenue is considered in the forecasting horizon, this means that there is a 15% chance that this opportunity is not included in the model since it is pushed out past our five-year range. This is similar to treatment of category E opportunities that do not have a financial impact within the immediate five years that the revenue is forecasted. Figure 13: Probability matrix of planned year versus actual start year. Planned Year Probability % 80% 15% 5% Domestic Actual Year % 75% 20% 5% % 70% 20% 10% % 65% 20% 15% % 60% 40% Planned Year Probability % 60% 25% 15% International Actual Year % 55% 20% 10% 5% 1000% % 50% 15% 5% 30% % 45% 10% 45% % 40% 60% For this model, inferences were made based on reasonable assumptions from core and ancillary markets. For core domestic markets, businesses are usually well informed on the timing and scope of the opportunity start dates and so high probabilities are assigned for the planned year. However, as the program lies further out in the timeline, there tends to be less certainty as the opportunity may not be mature

18 Forecasting Revenues in Ancillary Markets 17 enough to have a well-planned start year. For international ancillary markets, there is less certainty in the assumption that the planned start year is the actual start year. Whether that is an attribute of the market or lack of intimacy and knowledge of the market, higher uncertainty is generally the case. Similar to the domestic case, the probability for international opportunities also degrades as planned years moves further out in the timeline. It is noted that since international opportunities planned start years are less certain than domestic ones, more international opportunities than domestic opportunities will pushed out beyond the five-year scope of our revenue forecast. To provide some backup to these numbers, a study done on 18 major defense acquisition programs by SMA looked into the delay for both the requests for proposals and the program awards in recent contracts. 2 For the Request for Proposal (RFP) delays, there was a 78% probability it was awarded the same year it was planned, 17% within two years, and 6% within the three years. Similarly, for the program award delays, there was a 72% probability it was awarded within same year it was planned, 17% within the two years, 6% within three years, and 6% within four years. Taking the average of these delays gives roughly 75% probability that there is a delay within the year it was planned, 17% chance of delay within two years, 6% chance of delay with three years, and a 3% chance of delay within four years. These represent domestic program awards within the United States and the probabilities assigned to the probability matrix in Figure 13 are roughly within the same order of magnitude for the domestic opportunities. This is a good indicator that the numbers used are accurate representations of opportunity awards in core domestic markets. For international markets, it is well known these probabilities are less certain, so probabilities assigned are smaller than their domestic counterparts. 3 When running the scenario simulation, similar calculations as before are done. The reader is reminded that the static values in the pipeline did not change. The only change is that this iteration of the 1,000 scenarios is run with the added probability that the planned year is the actual start year. The percent difference formula is thus modified as follows % Diff = [B(t) B(go) B(win) - P(go) P(win)] R (4) where B(t) is the binary value assigned depending on whether or not the planned year is the start year. This value is assigned similarly to the other binary values using the probability values from Figure 13. Using Equation 4, the results are graphed and the percent differences are shown in Figure 14. There are some dramatic differences from this graph than that of Figure 12. The only similarity between them is that the mean percentage difference is less for domestic opportunities than the international ones and that as the years progress they get closer to 0% difference with smaller standard deviations. 2 The study done was a SMA funded project that analyzed the request for proposal (RFP) and award dates of major defense acquisition programs (MDAP). 3 Analysis of Major Defense Acquisition Programs (MDAP) Award Delays. SMA,

19 Forecasting Revenues in Ancillary Markets 18 Figure 14: Calculation of percent difference between expected value (EV) and scenario value (SV) with additional probability of timing delays added to scenario calculations Percent Difference (1 - EV/S) Domestic International Combined The percentage differences for both the domestic and international opportunities do not lie close to 0% difference. This implies that there is a systemic bias (over-estimation) of approximately 20% to 60% in the near-term forecast. In 2018 for the domestic case, the percentage difference is bigger than 20%. In the latter years, one can see that the percentage difference starts to converge close to the 0% axis meaning that the expected value results become closer to reality. The error bars on the domestic cases for the standard deviation are relatively small so there is little variability as before. As for the international case, there is a large percent difference of about 80% in 2018 with the years converging to approximately 18% in the later years. The more disconcerting result is that within one standard deviation a large percentage difference of up to 150% may be possible. Putting this into context, the revenue forecast can be as low as 10% or as high as 150%. This is clearly a dramatically erratic result and tells us that timing is highly important when considering forecasting revenues. Across the board, it is known that many companies have international revenue forecasting models that give drastically inaccurate results. From doing these simulations, it is seen how timing of the program start year is an attribute to factor into the expected value to develop a more accurate forecast.

20 Forecasting Revenues in Ancillary Markets Obstacles in Forecasting When entering new markets there are three underlying issues that create obstacles to forecast future revenues accurately: 1. Small Sample Sizes: The relatively small number of opportunities in new or ancillary markets is inherent with the pursuit of growth in these markets. Small sample sizes do not lend themselves to simple statistical methods that can be easily and intuitively incorporated in revenue forecasting. 2. Large Revenues: When entering a new or ancillary market, firms typically focus on projects with relatively large revenues to balance the pay-off against investment risk. These pursuits require significant investment of resources especially since the firm is not as familiar with the market and may be in a less advantageous competitive position. The large project revenue means that each opportunity is critical to the total forecasted revenue, thus outcomes can vary significantly on a single win or loss. 3. Timing: One of the most significant sources of uncertainty is timing. This is particularly true in public sector markets where projects are not necessarily driven by an urgent competitive situation and subject to government budgets and lengthy approvals. As seen from the results of Section 7, including a probability estimate for the planned start year in our testing creates drastically different results in estimating forecast accuracy. Each of these can have a significant impact on forecasting revenue techniques, specifically the expected value formula. For the first case, there exist statistical methods to handle small sample sizes. However, these statistics are difficult to implement as part of a routine business planning function and are nonintuitive. Secondly, being more selective about which opportunities to include or exclude from the forecast based on the static values assigned to the opportunity such as revenue and probabilities can be done. However, eliminating too many of these opportunities limits the sample size and introduces the problem inherent in the first case. See Appendix A for more details. Also, selectively excluding opportunities introduces additional biases since it involves non-bayesian estimation and is largely done either ad-hoc or with arbitrary rules. The third issue can be effectively addressed by modifying the traditional expected value formula with a probability estimate associated with the planned start date. This approach also helps mitigate the first two issues by help push out the revenues even more than the traditional expected value. Introducing an estimate of the probability that the actual start date is the planned start date is consistent with the Bayesian approach of the traditional forecasting methodology. The probability estimates can be informed and validated from historical data in the individual markets. Although each approach can improve the predictive capability of the model, our research indicates that addressing the timing issue with a new probability can significantly improve forecasts and is a simple enhancement to revenue planning process being used today at most firms. Our analysis also shows that small sample size statistics typically associated with pursuits in ancillary markets only account for a small percentage of forecast inaccuracy when compared to addressing the timing issue. See Appendix B for more details.

21 Forecasting Revenues in Ancillary Markets The New Expected Value Formula The introduction of an estimate of project timing uncertainty (i.e. the probability that the actual start year is the planned year) will be added to improve the traditional expected value-based forecasting approach. This requires the need to define a matrix that distributes the expected value based on probabilities of planned start year versus actual start year. This can be simplified by using a single probability value from which subsequent values in the matrix are derived. Since the total revenue of the opportunity is already distributed throughout the five years, the probability associated with timing to each of these five years is assigned and revenues are pushed with into the later years based off the probabilities. This creates the matrix redistribution of revenue that is used to create the New Expected Value (NEV) formula. Modifying the expected value formula from Equation 2, the formula is written as NEV[t i ] = 5 k=1 ( P(go) P(win) R[t i ] M ik [P(time), t i ]) (5) where i is the planned year, k is the actual year, P(time) is the probability that the planned year is the start year and M[P(time), t ] is the matrix associated with the timing of the opportunity. This formula redistributes out the revenue throughout the five years based on the individual timing probability. Equation 5 is abstract so an example will be shown to illustrate the type of matrix used in the new expected value formula. Figure 15shows an example of how revenue for a project is distributed throughout each year (for both the actual start year and the planned start year). This demonstrates the how the expected value (EV) and the new expected value (NEV) differ. The yellow shaded section represents the matrix M and the probability associated with it. Figure 15: Example pipeline of how revenue is distributed based on expected start year Opportunity (j) Actual Start Year (i) Planned Year (k)* >2023 Rev Potential [$100M] [$150M] [$150M] [$150M] [$125M] [$0M] [$675M] P(go) [90%] 90% P(win go = 1) [90%] 90% P(t) [80%: $65M] (1 P(t))*0.8 [16%: $13M] P(t) [80%: $97M] (1 P(t))*0.2 [4%: $3M] (1 P(t))*0.8 [16%: $19M] P(t) [80%: $97M] (1 P(t))*0.2 [4%: $5M] (1 P(t))*0.8 [16%: $19M] P(t) [80%: $97M] (1 P(t))*0.2 [4%: $5M] (1 -P(t))*0.8 [16%: $19M] P(t) [80%: $81M] (1 P(t))*0.2 [4%: $5M] (1 P(t))*0.8 [20%: $20M] Total [$81M] [$123M] [$123M] [$119M] [$101M] [NEV] [$65M] [$110M] [$119M] [$121M] [$105M] [$25M] [$547M] [EV] [$81M] [$122M] [$122M] [$122M] [$101M] [$0M] [$547M] [X] example calculations * For each planned start year, the matrix is displaced by one year down and one year to the right This entire figure represents some opportunity j that has a planned start year of The rows represent the planned years and the columns represent the actual start year where P(t) is the probability that the planned year is the start year. Looking at the row for planned year 2018, it is seen that it has the values P(t), (1 P(t)) 0.8, and (1 P(t)) 0.2 for 2018, 2019, and 2020 respectively. When the terms are added, the resulting total is 1 indicating that the total revenue assigned to planned year 2018 is distributed out three years.

22 Percent Difference (1 - NEV/S) Forecasting Revenues in Ancillary Markets 21 Similarly, this occurs for all the years and some of the revenue is distributed to 2023 and beyond which is not included in our total five year forecasting range. Starting from the top, the revenue for the opportunity, the two probabilities, and the matrix columns of the revenue are pushed out from the timing matrix. If the planned year is the expected year, that is P(t) = 1, then only the diagonal entries of the matrix would remain giving the traditional EV. Also, if the opportunity does not start for another n years the entries would all be displaced n years down and n years to the right. The matrix shows that the new expected value is lower in revenue in earlier years. This is expected since a probability is introduced that delays the expected start year versus the actual start year resulting in revenues being pushed out to later years. The last column in the table provides the total revenue for all the years. Although the sums at the bottom right are equal, the NEV has $25 million allocated to 2023 and later years, which are outside the range of the five-year forecast. Another attribute to note is that the matrix is designed to only assume one probability P(t) and the probability was used to create a discrete drop in revenue across the entire matrix. This makes it so that a company only needs to assign one additional probability rather than design an entire matrix for the NEV. The delay in timing is inherently embedded in this matrix and is easy to implement for companies developing revenue forecasting in ancillary markets. The discrete drops in delay are simple, but more complicated and possibly continuous model designs can fit actual historical data or known timing delays. As done before, the 1,000 simulations are run once again and the percent difference is plotted using the modified version of the percent difference formula given as % Diff = [B(t) B(go) B(win) NEV] (6) where NEV was given by Equation 5. Running the simulations using Equation 6, the results are shown in Figure 16. Similar to Figure 14, the domestic percent difference is more accurate than the international percent difference. Figure 16: Percent difference between NEV and scenario value Domestic International Combined