How to Separate Risk from Uncertainty in Strategic Forecasting Christian Schäfer

Transcription

1 How to Separate Risk from Uncertainty in Strategic Forecasting Christian Schäfer Preview Christian Schäfer takes us on a tour through pharmacological forecasting of the market potential for a new drug. En route, we ll see how risk and uncertainty are defined and made operational. Christian also offers pertinent examples of graphical representations that are effective for instructing senior management about the uncertainty surrounding drug forecasts information that is crucial to executive decision making. 12 INTRODUCTION Uncertain outcomes are an inevitable and unpreventable aspect of any project, and this is particularly true in the pharmaceutical industry. Development projects for a new chemical entity (NCE) are generally associated with relatively low probabilities of success, at least compared to those in other fields. At the start of development, a project s success rate is only 4 to 7 percent, and the time span of a project from start of development to the potential launch of the product is 10 years or more an eternity to those in electronics or software, for example. According to historical data from the Pharmaceutical Benchmark Forum in the United States on drug development and its costs (Harpum, 2010), the total project-development investment for an NCE that does eventually reach the market is in the range of $1 billion. The actual costs of a successful project are usually significantly lower, but every successful project needs to cover the costs of multiple other projects that die before reaching the market. Commercial and technical uncertainties plague project development for an NCE. The competitive landscape the new product will face at launch introduces uncertainty in pricing and market access. There is uncertainty over whether a target product profile (TPP) can be achieved: the TPP defines the potential product characteristics and benefits, such as efficacy, safety, and formulation at launch. These factors evolve over time, affecting the forecasts of drug success or failure and therefore financial gain or loss. For all of these reasons and more, the challenges posed by strategic forecasting, which aims to show a picture of a project s prospects and risks, are formidable. RISK VS. UNCERTAINTY In forecasting the success of an NCE, it is important to distinguish project risks from uncertainties. In early product forecasting, risks are associated with an event that can happen in one of several clearly distinguished ways. A good example is the number of competitors a company will face at launch. At project inception, this number is unknown, but several competitive scenarios are possible. A company could be the first to market with no competitors for a few years, possibly because the competition s development projects have stalled or failed. Or it could be the case that the company launches second or third to market. Because the development projects of competitors might either fail or succeed,

2 the business case is at substantial risk with regard to the market conditions at the time of launch. As we discuss later in this article, risk will be addressed by scenario modeling. Uncertainty, in the context of new-product forecasting, arises over differences between the model input assumptions made to project the drug development outcome and the actual values that the model inputs take on. As will be explained in the next section, the main model inputs deal with class share, inclass share, patient compliance, and future price. Departures from the assumed values here can lead to sizeable differences in the forecasts of project success and profitability. To understand the impact of uncertainty associated with the assumptions made for the different inputs of the forecast model, Monte Carlo simulation is an effective technique. When risk and uncertainty are mistakenly intertwined, the result can be misinterpretation of forecasts and erroneous decisions. This article discusses the importance of clear definitions and classification of risk and uncertainty when creating and presenting forecasts on the commercial attractiveness of a new product. PATIENT-BASED FORECASTING MODELS A patient-based forecasting model is a popular tool for predicting the commercial attractiveness of early-stage pharmaceutical development projects when historic sales data are not available. Its starting point is epidemiological research. An epidemiological report supplies a market-volume projection of potential patients. To use hypertension as an example, all patients receiving antihypertensive treatment would be taken into account, including estimates on how this number might develop over the coming years. The resulting number defines the overall market opportunity. For most indications (the conditions the drug is designed to treat), there are several therapeutic options, called therapeutic classes. A forecast model must input assumptions Key Points A clear distinction between risk and uncertainty is fundamental when forecasting highly indeterminate outcomes, such as the market chances of a new drug. Risk refers to events that can happen in one of a few clearly distinguished ways, such as whether a new drug is first, second, or third on the market. Uncertainty arises from differences between the model input assumptions (regarding future price and patient compliance, among others) made to project the drug development outcome and the actual values that the model inputs take on. Each of these uncertainties can assume a range of possible values. Risk is best operationalized via scenario modeling of discrete alternatives, while the range of uncertainty can be modeled effectively with Monte Carlo simulations. Graphical summaries are best suited for conveying risk and uncertainty to senior management. Graphics can be created to emphasize different aspects of the uncertainty associated with drug forecasts. about the future market share of each therapeutic class. For already established therapeutic classes, secondary data is available for projecting the future development of class shares. However, to forecast a prospective product s share of a new, not-yet-launched class is clearly a greater challenge, and additional market research is helpful here. Following the class-share assumptions, additional assumptions must be made about: the NCE s in-class share within the therapeutic class; the patient s willingness to take the drug regularly (compliance); and the new product s price development over time. FORESIGHT 13

3 Table 1. Patient-Based Model Assumptions Drug treated patients (millions) Market share of therapeutic class Product share within class Base Min 0% 0% 0% 0% 3% 8% 11% 14% 15% 16% 17% 17% 17% Base 0% 0% 0% 0% 4% 10% 14% 16% 18% 19% 20% 20% 20% Max 0% 0% 0% 0% 6% 13% 18% 20% 22% 24% 26% 28% 28% Min 0% 0% 0% 0% 100% 90% 70% 68% 55% 50% 48% 48% 48% Base 0% 0% 0% 0% 100% 100% 80% 75% 65% 60% 58% 56% 56% Max 0% 0% 0% 0% 100% 100% 90% 80% 70% 65% 65% 65% 65% Compliant days of Base therapy Min Price per day ($) of Base treatment Max Point forecast for Output base case revenue The first column of Table 1 summarizes the key inputs into a simplified version of a patient-based forecast model. Next to each input, you see assumptions for the base, minimum, and maximum levels. By multiplying the base input values together, we receive the point forecasts for base revenue shown in the bottom row. The brand team for the NCE usually has a good understanding about the rough development of the different input values over time. Of course, the final values of class share, in-class share, compliance, and future price are uncertain. To represent this uncertainty, we need to specify a reasonable range of values for each input, from min. to max. The more uncertainty there is about the future development of a model s input factor, the wider the value range will be. Discussions by the brand team will elicit multiple opinions, and these will be reflected in the assumptions. Figure 1. Sales-Range Chart from Monte Carlo Simulations ASSESSING UNCERTAINTY BY MONTE CARLO SIMULATION Monte Carlo simulation (MCS) is a standard technique for analyzing the impacts of uncertainty associated with the input assumptions. Sam Sugiyama s Foresight article (2007) provides a valuable primer on MCS and how it can be used to enhance a forecasting model. MCS can be applied to show how variability in...(uncertain inputs) affects the model s forecasts and forecast errors. In short, MCS is a way, sometimes the most effective way, to model uncertainty in the factors that influence the forecast and thus to clarify the scope and nature of forecast error. (Sugiyama, p.30) In a Monte Carlo simulation, a value for each uncertain input is randomly selected and used to calculate an output value ( Revenue, in our example). The input values will lie within the MIN to MAX range specified (as in Table 1). This process is repeated thousands of times with different random input values, each iteration resulting in a revenue forecast. The thousands of forecasts are sorted from low to high and various percentiles are calculated, such as the 10th and 90th percentile. The percentiles help to describe the probability of reaching various revenue levels. Figure 1 provides an illustration of the results. 14

4 Figure 1 presents the simulated development of revenues over time. The black line represents the brand team s base case, which is founded upon the base assumptions of Table 1. The base case is enveloped by the 90th (green) and 10th (red) percentiles. The 10th percentile represents a 10 percent chance that sales might be lower than indicated in this line, whereas in 90 percent of the cases in this scenario, sales will be higher than what is displayed in the red line. The probability that the true sales will reach a point within the corridor between the green and the red lines is 80 percent. The closer the red and green lines are to the black one, the more certain the brand team is about the underlying assumption of the forecast. The analysis presented in Figure 1 demonstrates that, from 2015 on, positive revenues are most likely to accrue and reach a peak of $ million annually from 2017 to The uncertainty faced in the success of this new product is now made tangible for decision makers. The simulated mean line (blue) within Figure 1 represents a benchmark for the brand team s base-case assessment (black). A simulated mean that is systematically higher than the base assessment indicates that the brand team s base case could be considered as too conservative. A simulated mean that is below the base case implies that the brand team s opinion may be overly optimistic. When the black and blue lines are more or less at the same level, the forecast reflects a balanced and realistic picture of a product s commercial potential. SCENARIO PLANNING FOR RISK In contrast to input uncertainty, risk is associated with an event that can happen in a small number of distinct ways. For an NCE, there are two major event types that affect the commercial attractiveness of the new product: those related to the competitive market environment, and those that shape the product s profile (TPP). The combination of these two types of risk can be developed through scenarios. Figure 2 illustrates the Risk-Scenario Matrix. This matrix presents three different states for the market environment and the TPP. When it creates a forecast, the brand team decides which competitive situation will be the most likely to occur. This is labeled the base case. For market environment, the base case shown is some competitors. For TPP, the base case is one of a high product efficacy and moderate safety. Scenario I in the table is the occurrence of the base case for both event types. The brand team has decided that this scenario has a 40 percent probability. Further assumptions for the best case (Scenario II) and worst case (Scenario III) have to be determined by the brand team. In Scenario II, for instance, a better safety profile compared to the base case and only one other competitor within our class could be assumed. In the worst case (Scenario III), only moderate efficacy and issues with the product s safety and three in-class competitors could be considered. Both assumptions in the worst-case scenario have a negative impact upon the product s commercial opportunities. For a proper project evaluation, usually at least Scenarios I, II, and III will be calculated. Scenario I reflects the base case, which usually is the one with the highest probability. Figure 2. Risk-Scenario Matrix, with Assumed Probabilities for Each Scenario FORESIGHT 15

5 Furthermore, Scenarios II (best case) and III (worst case) are calculated in order to benchmark Scenario I for possible up and downside potential. The mixed Scenarios IV and V are optional. Additional scenarios could in principle be analyzed, such as the blacked-out ones in the matrix shown. In addition, it is possible to include other risk factors in a risk-scenario matrix, e.g., price per dose (low, moderate, high), administration profile (once per day, twice per day), and the number of competitors in the class. However, one should always keep in mind that, as we examine proliferating scenarios, the payoff in terms of additional insight may not be worth the effort. How much more would we learn from considering a base market environment and best TPP scenario compared to Scenarios II and V? Does examining this additional scenario represent a justifiable use of our resources? The brand team needs to provide an estimate for the probability of occurrence for each of the defined scenarios. In their Foresight article, Nick Guthrie and Des Markland (2010) note that it helps to tell participants not to worry about being too precise in their probability assessments but [to] ask them to express probabilities in figures rather than words in order to limit ambiguity. Depending on a product s current development phase, it may not always be possible to Figure 3. Peak Sales-Range Chart, Showing Base Forecast, 10th and 90th percentiles differentiate scenarios for the TPPs. For start of development projects, often the same TPP is assumed for all competitors, since it is too early to make informed assumptions about TPP differentiation. Now that we know how to deal with uncertainty (through Monte Carlo simulations) and risk (through scenarios), it is time to put everything together. For each scenario, we create one model-input-assumptions worksheet, as in Table 1. The assumptions on the inputs in the worksheet will depend on the scenario s assumptions. For instance, when we are considering a scenario with many competitors, the min., base, and max. projections of the in-class market share will all be lower than in a scenario with fewer competitors. We then run the Monte Carlo simulations based on the worksheet for every scenario. The resulting base forecasts are already indicated in Figure 2. However, we still need a way to convey the variation in each scenario s Monte Carlo results, especially to the powers that be in the executive suite. Graphical methods are an excellent way to summarize this information, for a number of easily understandable reasons. GRAPHICAL FORECAST PRESENTATION It s important to utilize a means of presenting to senior management the forecast s inherent risks and uncertainties in a way that is transparent and quickly comprehendible. Fortunately, we have various graphical options at our disposal to accomplish this. Figure 1, the Sales-Range Chart, is useful to assess the uncertainty of a forecast scenario. The Risk-Scenario Matrix (Figure 2) gives a top line summary about the assumptions, probabilities, and peak sales for each standalone scenario. Peak sales in general are a key figure to anchor a forecast scenario. Figure 3 is a Peak Sales-Range Chart that comprehensively compares the simulated sales revenue results for the different scenarios. The upper and lower ends of each bar represent the 90th and 10th percentiles of the 16

6 Monte Carlo simulation results for the peak year. You can see that the uncertainty underlying Scenarios I, III, and V is lower than the others. Comparing Scenarios II and IV, the Peak Sales-range Chart clearly visualizes that Scenario II faces a massive downside threat, whereas Scenario IV shows limited downside potential but a large upside opportunity. Making the degrees of uncertainty transparent in this way is an invaluable aid for senior management. Figure 4 is a 3-D enhancement of Figure 3, in which the size of the bubble represents the point forecast for peak sales. The more probable a scenario is considered to be, the farther to the right the bubble is located. The more uncertain the payoff, the lower the bubble is located, where the range is again the delta between the 10th and 90th percentiles of the peak sales year. A position in the upper right would reflect a more probable scenario and one with a more certain payoff. We can see here that Scenario II, for example, offers the highest revenue forecast (largest bubble) but a high range of uncertainty in this payoff and relative low probability of occurrence (situated to the left). The graphical outputs presented up to this point have focused on peak sales and the related overall uncertainty. The Tornado Diagram of Figure 5 is a tool to present the sensitivity of a sales prediction to variation in input assumptions. Each row represents the uncertainty associated with one input source of the model (as listed in Table 1). In this example, the variation of sales revenue for the year shown is most sensitive to the assumption for class share, and insensitive to the assumed values for in-class share. For the class-share input, the base level (in Scenario I) for the year 2022 is 20 percent. If this assumption is exactly correct, the sales forecast would be $754 million. Since the brand team is not very confident about the class-share forecast, they Figure 4. Peak Sales: Uncertainty & Probability Cross-Grid Probability of Scenario defined a range between 17 and 28 percent. With the other inputs set to their base values, the chart shows a huge upside potential of +8 percent and a small downside potential of -3 percent for the class-share value. The tornado diagram has translated uncertainty regarding the class share in 2022 into a sales-range figure. If a number close to the max. of 28 percent becomes reality, the sales would end up at $980 million, while an inclass share of the min. of 17 percent would bring us to only about $640 million. A tornado diagram usually ranks the impact factor with the highest uncertainty at the top and cascades down to the uncertainty with the lowest impact on the forecast. The total lengths of each of the red and green bars in Figure 5 reflect the overall uncertainty Figure 5. Tornado Diagram for Sales in Year 2022: Scenario I FORESIGHT 17

7 regarding a certain parameter in U.S. dollars. In a proper tornado diagram, 5 to 15 sources of uncertainty are analyzed. The tornado diagram provides a quick overview of the leverage effects of particular uncertainties in a forecast scenario, and the associated upside or downside effects. CONCLUSION We have seen how to treat risk and uncertainty in new-product forecasting, especially in pharmaceutical forecasting. However, the question remains as to whether we really need to separate risk and uncertainty so strictly. Could we not simply model risk as one more numerical input, assign probabilities (e.g., to there being zero, one, or two competitors), and feed these probabilities into our Monte Carlo simulations? In principle, there is nothing to stop us from doing this. However, in the pharma market, one always has a particular competitor in mind when discussing risks. Whether or not this competitor manages to enter a particular market segment is a game-changing occurrence, with consequences on the entire subsequent go-to-market strategy. Simply mixing this alternative in with the other Monte Carlo simulations would divert attention away from this fundamental dichotomy that needs to be explicitly addressed. A careful distinction between risk and uncertainty helps to send clear messages to senior management in support of their decision-making processes. When Christian Schäfer is Director of Global Forecasting & Strategic Process at Boehringer Ingelheim GmbH, Germany. Christian holds a PhD in business administration and has recently published articles on drugmarket forecasting by the use of analogues as well as on performance measurement in forecasting. christian_hans.schaefer@boehringer-ingelheim.com interpreting a patient-based forecast, it is important to keep in mind that the range output for each scenario reflects the overall uncertainty associated with the defined market and product conditions for this scenario only. The project s overall risks and opportunities emerge when comparing the different scenarios. Each scenario needs to be clearly classified regarding competition and TPP. To analyze the overall risk of a development project s forecasts, one needs to compare the different scenarios peak sales levels while bearing in mind the probability of each scenario. Figure 4, the Uncertainty & Probability Cross- Grid, best supports this trade-off analysis. For any forecasting model, a Monte Carlo simulation will only be as good as the inputs from the brand team in setting up the model. And it is important to remember that the simulation represents probabilities, and not certainties. Nevertheless, MCS is a valuable tool when forecasting an unknown future. Through MCS and risk scenarios, senior management s decision-making processes will be based on an informed and comprehensive overview of risks and uncertainties associated with a project. A well-structured and transparent presentation of forecasts generates confidence and respect for the development project s forecasts and the work of the brand team. REFERENCES Guthrie, N. & Markland, D. (2010). Assessing uncertainty in new-product forecasts, Foresight, Issue 16 (Winter 2010), Harpum, P. (2010). Portfolio, Program, and Project Management in the Pharmaceutical and Biotechnological Industries, John Wiley & Sons, Inc., New Jersey. Knight, F. (1921). Risk, Uncertainty and Profit: Schaffner & Marx, Houghton Mifflin, Boston. Sonnenberg, F.A. & Beck, J.R. (1993). Markov models in medical decision making: A practical guide, Medical Decision Making, 13 (4), Sugiyama, S. (2007). Forecast uncertainty and Monte Carlo simulation, Foresight, Issue 6 (Spring 2007),