The Analytical Approach to Decision Making. Defense Resources Management Institute Naval Postgraduate School Monterey, California

Transcription

1 The Analytical Approach to Decision Making Defense Resources Management Institute Naval Postgraduate School Monterey, California

2 Analysis The process of breaking a complex topic or problem into smaller parts to gain a better understanding of it 2

3 EXTERNALITIES EXTERNALITIES TECHNOLOGICAL Strategic Planning Process Determine Goals, Policies and Objectives Selection of Programs LEGAL Implementation Process Develop Programs to Pursue Given Objectives Systems Analysis Program Analysis Operating Process Operations Analysis Specific Constraints Mixed Constraints General Constraints INPUTS Economy OUTPUTS Efficiency CULTURAL Performance Standards Develop Organizational Structure POLITICAL HISTORICAL Plan Organizational Structure PSYCHOLOGICAL ECONOMIC 3

4 Example: Base Closures Decision to close bases, and criteria for selecting bases. Systems Analysis Program Analysis Choice of bases, estimates of costs. Operations Analysis Planning and scheduling. 4

5 Environment Technological Economic Political Legal Psychological Cultural Historical Threats Systems Analysis Strategic Planning Process Determine Goals, Policies and Objectives Program Analysis Select Programs Implementation Process Develop Programs Within Given Objectives Select Forces, Budgets Operating Process Operations Analysis Use budget and forces Achieve performance objectives Alternatives Constraints National Security 5

6 McNamara (1961) Major decisions should be made by choices among explicit, balanced, feasible alternatives The Secretary should have an active analytic staff to provide him with relevant data and unbiased perspectives Open and explicit analysis, available to all parties, must form the basis for major decisions 6

7 Elements of Analysis GOAL(S) what is to be achieved OBJECTIVE(S) what to pursue to achieve the goal(s) ALTERNATIVE(S) MODEL(S) PREFERENCE(S) the potential solutions/policies/options we could choose to achieve the goal(s) the means by which we estimate, measure, or evaluate the consequences of an alternative rules for ranking the alternatives 7

8 The Process of Analysis FORMULATION (conceptual phase) define issues of concern, clarify the objectives, limit the problem SEARCH (research phase) develop alternatives, look for data, identify important relationships EVALUATION (analytic phase) build models; use them to predict consequences of alternatives INTERPRETATION (judgmental phase) compare alternatives based on outcomes predicted by the models, derive conclusions, indicate course(s) of action 8

9 Question Assumptions Define Issues of Concern Clarify Objectives Evaluate & Decide Collect Data ITERATION Test for Sensitivity Identify Alternatives Compare Outcomes Predict Consequences Build Models 9

10 Example: SAC Air Base Study Question Assumptions Deterrence Maximize Deterrence Evaluate & Decide Logistics Minimize Cost Collect Data Collect Data Test for Sensitivity Compare Outcomes Predict Consequences Build Models Identify Alternatives Identify Alternatives Predict Consequences Build Models 10

11 When Do You Stop? Question Assumptions Define Issues of Concern Clarify Objectives Evaluate & Decide If assumptions are still valid, stop here! Collect Data Test for Sensitivity Identify Alternatives Compare Outcomes Predict Consequences Build Models 11

12 Formulation What is the REAL problem? What is the RIGHT question to ask? What is RELEVANT to consider? What scope? Wide? Narrow? What do we include? What are the boundaries? 12

13 Everything Begins With the Need to Solve a Problem Whenever there is a Difference between what can be done and what you want to do, AND you do not know how to achieve the desire, there is a Problem. For each problem there may be several needs. A need is a condition requiring relief. 13

14 Why? What is the problem? Why do you want or think you need X? Why do you say that? Peel away the onion with a series of WHY? What aspect of the current situation is unsatisfactory? A problem statement is a clear description of the issue(s) that need to be addressed by at least one stakeholder. It is not a lack of a solution statement! 14

15 Mental Processes Typical of Defining Problems Indifference (1) Onset of Interest (2) Disappointment (6) Intense Ideation (7) Hmmm (4) Absorption (3) Analytical Evaluation (5) Onset of Frustration (9) Fatigue (10) Idea Generation Hybridization and Evaluation (8) Aha! (12) Incubation (11) Intuitive Skills time Analytical Skills 15

16 A Problem is Scoped by its Boundaries Boundary conditions are solution constraints Affecting ways in which objects interact All of the attributes ascribed to the solution to your problem must be contained within the domain of interest Conversely, if the boundaries are too limiting, the solution lies outside of the controllable region and is beyond reach of your efforts Defining the boundaries of the problem means first attempting to define the problem 16

17 Exploring System Boundaries One way to explore system boundaries is to model what you know within the context of the adjacent systems, showing the interactions (e.g., information flows that occur between them) These interactions will help define the boundaries of your system and identify events that will trigger responses required from your system (and others) 17

18 Infection Control Healthy Soldiers Transmission Natural Diseases Hospital Care Bioterrorism Sanitation Treatments Detection Development 18

19 Need Want Primitive needs are unsupported opinions of people involved with the system Effective needs are: Supported by organized evidence That evidence is tied to convincing rationale Someone must be willing to incur the cost & risk to fill it! 19

20 What is Needs Analysis? Initial Problem Statement (Primitive Need) From client or decisionmaker I need a bridge across the river Needs Analysis Revised Problem Statement (Effective Need -or- Capability Need) MORE BROAD & SOLUTION- INDEPENDENT I need some means of getting across the 20 river

21 Why do a Needs Analysis? Provides justification for proceeding with work cost vs benefit Cornerstone on which subsequent decision processes are built Goals, Objectives, Requirements Values, Priorities Inadequate Needs Analysis leads to an incorrect solution 21

22 Stakeholder Analysis Identify Relevant Stakeholders Clients (Sponsor, Decision-Maker, others) Users (Operators, Customers, Contractors, others) Analysts (Your design team, other experts) Others with a vested interest / knowledge of problem Develop a List of Questions / Desired Information Stakeholder Needs, Wants, Desires System Objectives Constraints Other pertinent information (Facts/Assumptions) Conduct Research, then Interviews Consolidate Information Gained Repeat as necessary 22

23 Who are the stakeholders? One perspective: those who are at risk if some aspect of the lifecycle of the system goes awry, causes harm, results in more losses than were anticipated or expected Another perspective those who have invested resources into solving the problem Not all stakeholders are created equal! 23

24 I was just wondering if you could help me find my way. Well that depends on where you want to get to. -Lewis Carroll,

25 Importance of Structure Identifying and structuring objectives is the single most important thing you can do in formulating a decision problem What do you need? What do you want? Not a question of what already exists If it worth doing a formal analysis of a decision problem, it is worth spending serious effort on identifying objectives 26

26 Goal Necessary to define a target A desired level of achievement It is either reached or not reached Can be answered with a yes or no Examples Eliminate improvised explosive devices (IED) Acquire surveillance capability yielding two days warning time 27

27 Discovering Your Goal(s) Descriptive scenario The way things are COMPARE Needs Prescriptive scenario The way things should be Goals Objectives 28

28 Objective Necessary to measure how close an alternative is to hitting the target How do you wish to get to the target Usually stated in terms of minimize or maximize Examples Maximize number of IEDs detected Minimize IED production Maximize reliability of radar Minimize commute time 29

29 Identifying Objectives: First Steps Start with strategic objectives: review planning and strategy documents Identify appropriate stakeholders and involve them in the process Decision makers Superiors or commanding officers Other leaders Operators or customers Community leaders Government agencies Legislative bodies 30

30 Generating Possible Objectives Expansive generation of objectives (pruning and structuring comes later) Solicit others ideas but avoid group work initially Brainstorming: solicit objectives from stakeholders without evaluating them You may or may not want stakeholders in a room together Try to focus on objectives not positions or alternatives 31

31 Some Questions to Generate Objectives What is your goal? Why is there a decision to be made? If you had no limitations or constraints, what would your objectives be? What is your ideal outcome of the decision, and what makes it so ideal? What is your nightmare scenario and what makes it so bad? What consequences would be unacceptable? 32

32 Search What are the ALTERNATIVES? What DATA do we need? What are the important RELATIONSHIPS? 33

33 Finding Alternatives All courses of action available to you Find an appropriate set of possible alternatives to examine Too few -> may not find the best one Too many -> may reduce quality of analysis 34

34 Alternative Formulation is a Systems Process That Proceeds From the General to the Detailed Alternatives should initially be presented as general concepts Estimates of impacts on objectives should be made Pros and cons of each alternative should be identified Infeasible alternatives should be discarded Promising alternatives should be analyzed in greater detail Adapted from 35

35 Alternative Formulation is an Iterative Process Relaxing Constraints Objectives Brain Storming Screening Feasibility Alternative Formulation Resisting Point Designs Gaining New Perspectives Considering the Improbable Adapted from 36

36 FUTURE CONDITIONS The Environment PREFERENCES Goals and Objectives Decision Result Value ALTERNATIVES OUTCOMES PAY-OFF 37

37 MODELS... Models are abstract representations of reality which help us to perceive significant relations in the real world In no case are models photographic reproductions of reality; if they were, they would be so complicated that they would be of no use to us. They have to abstract from a great deal of the real world -- focusing on what is relevant for the problem at hand, ignoring what is irrelevant. Hitch and McKean Economics of Defense in the Nuclear Age,

38 Role of Models Real World Real Behavior Abstract Theoretical World Model Analyze Observe Real Conclusions Interpret Model Conclusions 39

39 Types of Models All Models Mental Models Symbolic Models Physical Models Mathematical Models Verbal Models 40

40 Mental Models If I decide to choose that alternative...then I expect this as a consequence. 41

41 Mathematical Models (The Elements) VARIABLES - represent things that change as when the decision alternative changes and/or the future conditions change. RELATIONS - represent the connections between variables. PARAMETERS - represent the assumptions and information available from data. 42

42 Mathematical Models (Forms of Representation) 1. Algebraic - using equations y = a + bx 2. Graphical - using pictures y x 3. Tabular - using tables of numbers x y

43 Model Building No Simplify No Simplify Identify the problem Describe the problem (Assumptions) Can you formulate the model? Build Model (Assumptions) Can you solve the Model? yes Start Test the Model Finish Implement and use the model yes Is the model useful? No Refine 45

44 Rental car Miles Driven Mileage rate price Weekly mileage cost Alternative Weekly Flat fee Weekly Rental cost 47

45 Consider three alternatives for a rental car: Avion: Hurts: Bottom$: Alternatives Choosing a Rental Car $70 per week plus $0.30 per mile. $100 per week with free mileage for the first 200 miles, then $0.80 per mile. $160 per week with free mileage. Which is the best choice? 48

46 Building the Mathematical Models ALGEBRAIC MODEL: - A: Cost = (mileage) - H: Cost = if mileage <= 200 then cost =100 else cost = (mileage - 200) - B: Cost =

47 Building the Mathematical Models TABULAR MODEL: MILES AVION HURTS BOTTOM DOLLAR

48 Weekly Auto Rental Cost $300 $250 $200 Cost $150 $100 $50 $ Mileage Avion Hurts Bottomdollar 51

49 Model Building Decision rule becomes clear - when to switch Reduces problem to its most important element - mileage Aids in understanding the influence of parameters What happens if the flat rate on Bottom Dollar drops to $120? What happens if the milage rate on Hurts is reduced to $

50 Model Complexity A model always simplifies reality. Incorporate enough detail so that: Results meet your needs Model is consistent with available data Model can be analyzed in the time available 53

51 Error Model Complexity Optimal complexity depends on the decision context! Total Oversimplification Measurement Complexity 54

52 Everything should be made as simple as possible - but no simpler Albert Einstein

53 FUTURE CONDITIONS The Environment PREFERENCES Goals and Objectives Decision Result Value ALTERNATIVES OUTCOMES PAY-OFF 58

54 Decision Environments Certainty Uncertainty Complete information Incomplete information 59

55 as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know Secretary of Defense Donald H. Rumsfeld 60

56 Decision Environments Certainty Uncertainty You know: all the alternatives the one future condition all the outcomes 61

57 Decision Matrix Under Certainty Future Condition Alternative 1 Alternative 2 : Alternative 3 62

58 Decision Environments Certainty Uncertainty Complete information Incomplete information You do not know one or more of: all the alternatives all the future conditions all the outcomes all the probabilities of the future conditions 63

59 Decision Environments Certainty Uncertainty Complete information Incomplete information You know: all the alternatives all the future conditions all the outcomes all the probabilities of the future conditions 64

60 Static Decisions Decision Matrix Future Condition 1 Future Condition 2 Future Condition N Alternative 1 Probability Alternative 2 Outcome Alternative K 65

61 Probability A measure of the likelihood of the occurrence of a future event The quantification of uncertainty 66

62 Histogram

63 Relative Frequency Earthquake Problem Do Nothing New Codes Implementation Cost 0 10 F 1 F 2 F 3 F 4 F 5 F 6 F 7 F 8 New Codes & Retrofit 50 68

64 Relative Frequency Earthquake Problem Do Nothing New Codes New Codes & Retrofit Implementation Cost F 1 F 2 F 3 F 4 F 5 F 6 F 7 F ,500 25, ,000 2,500,000 25,000, ,106 32, , ,214 23,780 2,097, , ,000,000 16,777,226 1,334,889 69

65 Relative Frequency Earthquake Problem Decision Rule: Most Likely Do Nothing New Codes New Codes & Retrofit Implementation Cost F 1 F 2 F 3 F 4 F 5 F 6 F 7 F

66 Relative Frequency Earthquake Problem Decision Rule: Expected Value Do Nothing New Codes New Codes & Retrofit 411,592 32,030 2,730 71

67 Malaria Prevention Mental Map Exposed to Malaria? Take Malaria Pills? Payoff 72

68 Malaria Prevention (Decision Matrix) F 1 Exposed to Malaria F 2 Not exposed to Malaria Take Malaria Pills Don t Take Malaria Pills 73

69 Dynamic Decisions What happens when choice of an alternative changes the matrix? when the likelihoods change? when the number of future conditions change? when there is a sequence of decisions? 74

70 Tree Model NODES BRANCHES Decision node Future condition node 75

71 Malaria Prevention (Decision Tree) Payoffs Take Malaria Pills Exposed to Malaria Not Exposed to Malaria Don t Take Malaria Pills Exposed to Malaria Not Exposed to Malaria 76

72 Why use a decision tree? (Sequential decisions) If you have been exposed to malaria, you can take medications immediately after exposure to prevent malaria Lowers your chances of malaria 77

73 Malaria Prevention Mental Map Exposed to Malaria? Take Post- Exposure Pills? Develop Malaria? Take Malaria Pills? Payoff 78

74 Malaria Prevention Malaria Take Post- Exposure Pills No Malaria Take Malaria Pills Exposed to Malaria Not Exposed to Malaria No Post- Exposure Pills Malaria No Malaria Malaria Don t Take Malaria Pills Exposed to Malaria Not Exposed to Malaria Take Post- Exposure Pills No Post- Exposure Pills No Malaria Malaria No Malaria 79

75 Which Alternative is Better? Benefits B A C $ Cost 80

76 Which Alternative is Better? Benefits A C B $ Cost 81

77 Which Alternative is Better? Benefits A C B $ Cost 82

78 Decision Criteria With UNLIMITED resources if there is any positive benefit, then no matter what it costs, make the investment in: National Security; Transportation; Education; Health & Safety Regulations Posner & Adler (Eds) 2001 Cost-Benefit Analysis 83

79 Why Worry About Cost? Any course of action, any decision, will exact a cost Cost is a measure of the consequences of our decision As long as resources are limited, cost will be a factor in our decision 84

80 Decision Criteria Equal Benefits Minimize costs Equal Costs Maximize benefits Different Costs and Benefits If benefits can be monetized Net Present Value If not Need to make tradeoffs! 85

81 Getting Started Identify feasible, mutually exclusive alternatives Define the planning horizon Develop cash flow profiles Specify the interest rate to be used 86

82 Planning Horizon The period of time over which the cash flows of alternatives are compared Must be the same for each alternative Must consider useful life of alternatives Common approaches Least common multiple of lives Shortest life Standard horizon 87

83 Specifying the Interest Rate Circular A-94 sec 8 Benefit-Cost Analysis Benefits and costs can be monetized Real 7% (marginal pretax rate of return on an average investment in the private sector) Market interest rates are nominal Cost-Effectiveness Analysis Most DoD decisions fall in this category Treasury s borrowing rates for comparable length of maturity Published Treasury rates are nominal 88

84 OMB Guidance Cost-Effectiveness is appropriate whenever it is unnecessary or impractical to consider the dollar value of the benefits. Analysis of alternative defense systems often falls in this category. OMB Circular A-94, par. 5b 89

85 Cost-Effectiveness Analysis Benefits can not be quantified in monetary terms Define effectiveness based on desired capabilities/characteristics Measure capabilities of alternatives and assign a measure of effectiveness Identify "efficient frontier" Incremental analysis (tradeoffs) effectiveness vs. cost 90

86 Risk is part of life Every decision in an uncertain world involves some degree of risk. There is the risk that the cost will be higher than expected. There is the risk that the benefit will be lower than expected. 91

87 ASSESSING RISK What can go wrong? What is the likelihood? What are the consequences? What is my exposure?

88 What do we mean by Cost Risk? What can go wrong? The actual cost of a program exceeds the budget for that program (cost overrun) What is the probability of a cost overrun? What are the consequences of a cost overrun?

89 Normal Distribution Probabilities 68% X

90 Normal Distribution Probabilities 95% X

91 Normal Distribution Probabilities 99% X

92 Thinking about the budget Funding at the 50% level means there is a 50% chance of cost overrun.

93 Comparing the Risk of Alternatives P(cost overrun) = 9.63% P(cost overrun) = 97.76%

94 Acceptable Risk? Risk cannot be spoken of as acceptable or not in isolation, but only in combination with the costs and benefits that are attendant to that risk. Considered in isolation, no risk is acceptable! A rational person would not accept any risk at all except possibly in return for the benefits that come along with it. Even then, if a risk is acceptable on that basis, it is still not acceptable if it is possible to obtain the same benefit in another way with less risk. Kaplan and Garrick, On The Quantitative Definition of Risk, Risk Analysis, Vol. 1, No. 1,

95 Risk Tradeoff Risk Too Risky Acceptable Risk Proposed Budget Required Budget 100 Cost

96 Risk Management Risk old new Cost 101

97 Risk Management Risk Less Risk new Proposed Budget Cost 102

98 Risk Management Risk Acceptable Risk new Less Cost Cost 103

99 Ultimately, policy makers must decide how much the United States is willing to pay to lower the risks associated with deploying forces abroad. But some might argue that defense planners occasionally focus on absolute requirements the minimum number of forces that they believe will meet DoD s military needs without fully weighing the relative risks and costs of alternative levels. Moving U.S. Forces: Options for Strategic Mobility Congressional Budget Office, Feb

100 Pitfalls in All Analysis Not enough time spent defining the problem. Examining a restricted range of alternatives. Too much time spent in the details of the models. 105

101 Advantages of Analysis Answers are accessible to critical examination Answers can be retraced by others Answers can be modified by others 106