Decision Support for Leaders Level III DoD FM Certification

Transcription

1 Decision Support for Leaders Level III DoD FM Certification LTC Steven G. Hanson Defense Resources Management Institute Naval Postgraduate School Army G1 Plans and Resources Pentagon 1

2 Learning Objectives 1. Understand concepts underlying good decision analysis 2. Appreciate pre-decision analysis tools a. Cost-estimating b. Measuring effectiveness 3. Appreciate post-decision analysis tools a. Cost accounting b. Performance management 4. Specify, collect and use data 5. Be familiar with pre and post decision cross cutting accounting tools 6. Be familiar with sensitivity analyses and regression 2

3 1. Making Good Decisions The most important step is to formulate the problem correctly and completely What is the question to be answered? What are the exact goals and objectives? Who needs to be involved? What is the scope of the problem? What are the alternatives? What data do you need to find? 3

4 I was just wondering if you could help me find my way. Well that depends on where you want to get to. -Lewis Carroll,

5 Military Decision Making Process Receipt of mission Mission analysis Course of action development Course of action analysis Course of action comparison Course of action approval Orders production **Decision support in military jargon**

6 The Process of Analysis FORMULATION (conceptual phase) define issues of concern, clarify the objectives, limit the problem SEARCH (research phase) develop alternatives, look for data, identify important relationships EVALUATION (analytic phase) build models; use them to predict consequences of alternatives INTERPRETATION (judgmental phase) compare alternatives based on outcomes predicted by the models, derive conclusions, indicate course(s) of action 6

7 Formulation What is the REAL problem? What is the RIGHT question to ask? What is RELEVANT to consider? What scope? Wide? Narrow? What do we include? What are the boundaries? 7

8 Elements of Analysis GOAL(S) OBJECTIVE(S) ALTERNATIVE(S) MODEL(S) PREFERENCE(S) what is to be achieved yes or no THIS IS BINARY what to pursue to achieve the goal(s) (Maximize.. Minimize ) the potential solutions/policies/options we could choose to achieve the goal(s) the means by which we estimate, measure, or evaluate the consequences of an alternative rules for ranking the alternatives 8

9 Stakeholder Analysis: Power vs. Interest Grid High Subjects Players Interest Low Low Crowd Power Context Setters High Eden, C. and Ackerman, F. Making Strategy: The Journey of Strategic Management, 1998, p.122 9

10 Search What are the ALTERNATIVES? What DATA do we need? What are the important RELATIONSHIPS? 10

11 Evaluation - Role of Models Real World Real Behavior Abstract Theoretical World Model Analyze Observe Real Conclusions Interpret Model Conclusions 11

12 Model Construction No Simplify No Simplify Identify the problem Make Assumptions Can you make the model? Yes Can you Solve the model? Yes Enter Test the model Maintain the model Implement the model Yes Is the model useful? No 12

13 Interpretation Agree upon alternatives Feasible Acceptable Sustainable If they don t meet minimum criteria, kill em!! 13

14 2. Pre-Decision Analysis The analysis you conduct to help you make choices among alternative courses of action Cost - Estimating Measuring Effectiveness 14

15 Decisions in Defense Sector When you or other officials make a decision in the defense sector, what are your objectives? Max. firepower Min. cost Max. security Max. capabilities Min. number of deaths Min. Max. Max. skills energy Effectiveness usage 15

16 Choose the Best Alternative Choose the alternative that minimizes cost and maximizes effectiveness How can I know the cost of alternatives beforehand? How can I combine all my objectives into one measure of effectiveness? What should I do if one alternative is the cheapest and another alternative is the most effective? 16

17 Cost Estimating 17

18 What Is Cost? The real physical resources consumed The value of benefits forgone in the alternative use of resources Money equivalent of the real physical resources used 18

19 Total Cost of Alternative All relevant costs of the system Over the lifecycle of the system Based on expected operating conditions Having accurate and revealing cost information is critical for decision making. Understanding cost behavior is the key to useful cost analysis. 19

20 Lifecycle Phases Lifecycle cost includes all the relevant costs of an alternative Total lifecycle cost (LCC) Research and development (R&D) Procurement (investment, acquisition) Operations and maintenance (O&M) Disposal (decommissioning, retirement, salvage) 20

21 Cost over Time Procurement cost LCC O&M cost R&D cost Disposal cost 21

22 LCC for One Wind Turbine Year R&D Acquisition Military construction Total O&M Maintenance Civilian pay Disposal Total cost In millions of $ LCC for one wind turbine = $23.00 million 22

23 Cost Estimating and Budgeting Budget should include entire lifecycle costs How to estimate these costs? 1. Analogy (ratio) 2. Expert elicitation 3. Industrial engineering 4. Parametric estimation (linear regression) 23

24 Analogy (Ratio) Estimates new costs by comparing with old Identify a key characteristic Comparison gives an estimating ratio New estimate is product of old cost and cost estimating ratio Example Acquisition cost of old turbine = $1.8 mil New turbine has blades that are 20% longer Estimate of acquisition cost for new turbine = $1.8 mil * 1.20 = $2.16 mil Requires little effort but may be inaccurate 24

25 Expert Elicitation Expert s belief about the future costs Problem of overconfidence Estimated costs may be less than actual costs Experts may not consider what can go wrong that will increase the cost What if the contract award is disputed? What if the technology is unproven? What if the number to purchase changes? What if there is a supply shortage? 25

26 Industrial Engineering Identifies costs at all levels Requires complete list of resources for each task Documents Materials Labor Time Unit cost of each resource 26

27 Acquisition Costs for New Wind Turbine Component or subsystem Cost estimate 3 blades $640,000 Generator $430,000 Gear box $250,000 Tower $360,000 Base $235,000 Tools and test equipment $73,000 Transportation and assembly $124,000 Total acquisition cost $2,112,000 27

28 Statistical or Parametric Cost Estimating Very widely used cost estimating technique Availability of historical data for cost models (linear regression) 28

29 Time Value of Money What would you choose? Now One year from now or Preference for money now versus money in the future 29

30 Discount Factor Discount factor = 1 1+r n Number of years or periods r = 0.05 Discount rate Discount factor

31 Key Takeaways Total LCC = R&D + procurement + O&M + disposal Methods of cost estimation Analogy Expert elicitation Industrial engineering Parametric estimation Present value analysis to compare costs over time 31

32 Effectiveness 32

33 Defining Effectiveness Measures the extent to which an alternative helps to pursue objectives Takes into account relative importance of each objective Does not consider cost Relies on the decision maker s preferences Different people may determine that the same alternative has different levels of effectiveness 33

34 Three Questions to Define Effectiveness 1. What is important? Objectives hierarchy 2. How much is enough? Value functions 3. How important is it? Tradeoff weights 34

35 What Is Important? Building an objectives hierarchy Where do you start? Max. overall effectiveness What do you mean by that? Max. performance Max. availability 35

36 Example Green Initiative 5 alternatives wind power weatherizing solar panels green electricity electric vehicles 36

37 Green Initiative Objectives Hierarchy Max. overall effectiveness of green initiative Max. security of energy supply Min. harmful emissions from energy Min. chance of disruption of energy supplies Max. partnerships with industry to ensure energy supply Min. energy consumption Max. use of renewable or alternative energy resources When do you stop? 37

38 Green Initiative Objectives Hierarchy Max. overall effectiveness of green initiative Max. security of energy supply Min. harmful emissions from energy Min. chance of disruption of energy supplies Probability (percentage) Max. partnerships with industry to ensure energy supply Number of companies Min. energy consumption British thermal unit (BTU) per gross square foot Max. use of renewable or alternative energy resources Percentage of renewable resources 38

39 Objectives Hierarchy - Summary Identifying objectives is most important step in defining effectiveness Objectives seek to maximize or minimize something in order to achieve a goal Bottom level of the hierarchy should be measurable attributes Objectives may be derived from policy documents 39

40 Prioritizing Objectives 1. Maximize use of renewable or alternative energy resources 2. Minimize chance of disruption of energy supplies 3. Minimize energy consumption 4. Maximize partnerships with industry to ensure energy supply Wind power Green electricity 8% 6% 14% 4% 88,000 BTU per gross square foot 84,000 BTU per gross square foot 2 companies 4 companies 40

41 How is each attribute valued? Each attribute is translated to a value scale between 0 and 1 Determine Not enough performance value = 0 Good enough performance value = 1 Is more or less better? Assume a linear value function (easiest) 41

42 Example: Use of Renewable Resources Not enough renewables < 3% = x i L Enough renewables > 8% = x i H Consider an initiative with 6% renewables Calculation Percent x i L x i H x i L = (6-3)/(8-3) =

43 How Important Is It? Assess weights that describe the relative importance of one attribute versus another attribute Represent the tradeoffs a decision maker is willing to make between attributes Each weight is between 0 and 1 and sum of all the weights equals 1 Preferences of the decision maker 43

44 Tradeoff Weights Effectiveness Security Emissions Chance of disruption Partnerships Energy consumption Renewable resources 44

45 Effectiveness of Alternatives Calculate measure of effectiveness (MOE) for each alternative Alternatives: weatherize, solar panels, green electricity, electric vehicles, wind power Attributes: chance of disruption, partnerships, energy consumption, use of renewable resources We are using the same objectives hierarchy, value functions, and tradeoff weights for each alternative!!! 45

46 Measures of Effectiveness: Calculations MOE = (measureable attribute value of attribute weight of attribute)

47 Comparing Alternatives Why is green electricity more effective than electric vehicles? 47

48 Key Takeaways Three questions to define effectiveness What is important? objectives hierarchy How much is enough? value functions How important is it? tradeoff weights Preferences of the decision maker 48

49 Cost-Effectiveness 49

50 Cost Effectiveness Dominant solution Efficient set of alternatives 50

51 Choose the Best Alternative Choose the alternative that minimizes cost and maximizes effectiveness How can I know the cost Careful cost estimation of alternatives of overall LCC beforehand? How can I combine all my Answering three questions objectives into one to define effectiveness measure of effectiveness? Ask What decision should maker I do if if one he or she alternative is willing is to the spend cheapest more and money another to alternative gain extra is capabilities the most of effective? more effective alternative 51

52 Questions to Ask Decision Maker What is your budget for green initiatives? Do you have a required level of effectiveness? Do you want to spend $26 million more to gain 0.11 in effectiveness to select solar panels over weatherize? Do you want to spend $26 million more To use more renewable resources and have more partnerships with industry (advantages of solar panels vs weatherize) But greater chance of disruption and more consume more energy (disadvantages of solar panels vs weatherize)? Do you want to spend $36 million more to gain 0.25 in effectiveness to select green electricity over weatherize? 52

53 Key Takeaways Cost-effectiveness requires 1. Discounted LCC estimate for each alternative 2. MOE as measurement of extent to which each alternative attains objectives Alternatives that are less effective and more costly should be eliminated Best alternative depends on decision maker s preferences for minimizing cost and maximizing effectiveness Process is more important than the final number 53

54 Post-Decision Analysis The analysis you conduct to see how well you are doing Cost Accounting Performance Management 54

55 Cost Accounting A subset of managerial accounting dealing with relevant cost elements necessary for management analysis and decision making purposes. 55

56 Cost Accounting Uses Cost control Budgeting Performance measurement Determining reimbursements Setting fees and prices Program evaluations Economic choice decisions 56

57 Financial vs. Managerial Accounting Financial Managerial Users External Internal Rules GAAP SFFAS Scope (detail) Entire Org Parts of Org Frequency Qtr/Annual As needed Orientation Past Future 57

58 Performance Management and Focuses on: Accounting Data Improving effectiveness, focusing on how well desired outcomes are achieved; Improving efficiency, focusing on how well the costs of producing services and goods are managed; and Improving accountability, focusing on bringing together budgets and performance measures Time Quality Cost

59 Government Performance and Results Act of 1993 (Public Law ) Part of a series of laws in the 1990s aimed at improving federal government management Requires federal agencies to develop strategic plans with long-term goals performance plans with annual goals and measures performance reports on prior year performance Also requires a government-wide performance plan, intended to provide a single, cohesive picture of federal performance GPRA Modernization Act, Licht B., GAO, 2 Mar

60 GPRA Modernization Act of 2010 (Public Law ) The Act creates a new government-wide framework including long-term federal government priority goals revised federal government performance plan requirements quarterly priority progress reviews a government-wide performance website Performance.gov GPRA Modernization Act, Licht B., GAO, 2 Mar

61 Outcomes Threat Strategy Outputs Activities What information could we need to measure and manage performance? Forces Budget Inputs Organization Funds

62 What information could we need to measure and manage performance? End and Intermediate Outcomes (Effects, Results, etc.) Outputs Processes (Workload or Activities) Inputs Efficiency and Productivity Measures (use inputs and outputs) Workload Characteristics Explanatory Information Adapted from Hatry, H.P in Quicker, Better, Cheaper: Managing Performance in American Government.

63 Framework: A Performance Hierarchy Stakeholder s Desired End Outcome Focuses on how well desired end outcome is achieved Intermediate outcomes: Effectiveness of outputs (their quality, timeliness or other service characteristics) How well desired outcomes from internal processes are achieved Time Quality Cost Intermediate outcome: Efficiency How well the costs of producing services and goods are managed

64 Framework: A Performance Hierarchy Stakeholder s Desired End Outcome Focuses on how well desired end outcome is achieved CUSTOMER FOCUSED Intermediate outcomes: Effectiveness of outputs (quality, timeliness, service characteristics) Inter. outcome1 Inter. outcome n Intermediate outcome: Efficiency Output 1 Output 2 Output n MANAGEMENT (INTERNALLY) FOCUSED Inter. outcome 2 Activity 1 Activity 2 Activity n Output 1 Output 2 Output n Input cost 1 Input cost 2 Input cost n

65 Definitions Resource an input, cost recorded as line item expenditure Activity a task or set of tasks that use inputs and contribute to the production of outputs Process a set of interrelated activities that result in an output Output a product or service, the result of work done in a process or activity Cost object the output of interest for unit costing purposes Customers, products, services, programs, contracts or projects. 65

66 Characteristics of Good Performance Goals Quality over quantity Importance to budget decisions Public clarity Feasibility Adapted from Performance Measurement Challenges and Strategies, June 18,

67 Measuring Performance: Effectiveness You must know: The outputs The customers The desired outcomes What you want to do with the information 67

68 Example: Performance Hierarchy for U.S. Army G1 Army Desired Outcome: to fight and win our Nation's wars, by providing prompt, sustained, land dominance, across the full range of military operations and the spectrum of conflict, in support of combatant commanders. Intermediate outcomes: Army G1 Desired Outcome: Develop Army policy for all systems and functions in human resource support Intermediate outcome: Be efficient: Keep unit costs of processes and outputs as low as possible Plan Resource Manage Control Intermediate outcomes: Quality, time, and performance measures 68

69 Use a Performance Hierarchy to Measure Effectiveness Goal 1. Installation produces as much energy on-site as it uses Objective 1.1: Minimize the use of externally produced energy Objective 1.2:.. 69

70 Use a Performance Hierarchy to Measure Effectiveness Goal 1. Installation produces as much energy on-site as it uses Objective1.1: Minimize the use of externally produced energy Objective1.2: Electric Vehicle Initiative Measure: Gallons of fuel 70

71 Develop Targets for Measures Goal 1. Installation produces as much energy on-site as it uses Objective 1.1: Minimize the use of externally produced energy Objective1.2: Electric Vehicle Initiative Measure: Gallons of fuel Target: Use 20% fewer gallons of fuel next year 71

72 Measuring Efficiency Unit Cost A measure of efficiency the relationship between the cost of inputs and the amount of outputs. You must know: The cost of all the inputs How the outputs are produced (activities) Amount and kind of outputs 72

73 Calculating Unit Cost 73

74 Appropriations Appropriations Base Military Personnel $ 20M Operations & Maintenance $ 40M Procurement - Research & Development - Military Construction $ 50M 74

75 Base Activities Security Contracting Financial Management Human Resources Information Technology Administration 75

76 Base Expenditures Security Contracting Fin Mgt HR IT Admin Budget ($M) Civ Pers $ 15 Mil Pers $ 20 Equipment $ 10 Utilities $ 2 Comm $ 5 Supplies $ 5 Other $ 3 Relationship: Output = f(inputs) Total $ 60?????? 76

77 Base Expenditures Security Contracting Fin Mgt HR IT Admin % labor hours Budget ($M) Civ Pers $ Mil Pers $ Equipment $ Utilities $ Comm $ Supplies $ Other $ Total $ 60 $ 6.00 $ $ 4.50 $ 6.00 $ 3.00 $

78 Base Output Units Security Patrol hours Contracting Contracts managed Financial Management Accounts managed Human Resources Positions managed Information Technology Cases resolved 78

79 Develop Efficiency Targets Goal 1. Lower overall costs of operation by xx% Objective1.1: Minimize unit cost of IT cases resolved Objective1.2: Maximize xxxxx Target: Reduce unit cost of IT cases resolved by 5% next year Target:xxxx 79

80 Base Unit Cost Security Contracting Fin Mgt HR IT Admin % labor hours Budget ($M) Civ Pers $ Mil Pers $ Equipment $ Utilities $ Comm $ Supplies $ Other $ Total $ 60 $ 6.00 $ $ 4.50 $ 6.00 $ 3.00 $ Units 8,760 1, ,000 UNIT COST $ 685 $ 16,000 $ 15,000 $ 12,000 $ 60 80

81 Output or Management Budget IT cases resolved 50,000 cases x $ 60 = $ 3,000,000 quantity x price = budget A measure of workload 81

82 Side effects Outcomes include side effects, whether intended or not and whether beneficial or detrimental. If the program recognizes in advance that such side effects can occur, it should design the performance measurement process to regularly measure them. H. P. Hatry,

83 Bounded Targets A bounded target has two parts: The result desired (an objective), stated as a measurement The results that cannot happen (restrictions) a measurement + what s good enough + what must not happen Adapted with permission from Executive Learning Group, 83

84 From performance metrics to bounded target a measurement + what s good enough + what must not happen Bounded Target Reduce the unit cost of IT cases resolved by 5% Complaints on resolution of IT issues do not rise Random audit does not reveal an increase in errors in cases resolved Objective Reduce / minimize the unit cost of IT cases resolved Metric IT cases resolved 84

85 Bounded Targets assure vertical AND horizontal alignment Bounded Targets avoid situations where clear goals and motivation for one individual or organization have unintended negative consequences for others 85

86 Program Effectiveness The key to assessing program effectiveness is measuring the right things. Performance measures should capture the most important aspects of a program s mission and priorities. Appropriate performance goals should: 1) include both performance measures and targets; 2) focus on outcomes, but use outputs when necessary; and 3) include both annual and long-term measures and targets. Performance Measurement Challenges and Strategies, June 18,

87 The Role of Leadership Perhaps the most important aspect of any effective performance management system is ensuring active leadership engagement. Leadership engagement fosters a high-performance culture that empowers employees at all levels and enables the organization to work across silos to solve problems. In particular, the leader s responsibility not just for establishing the goals but also for conducting datadriven reviews are critical for creating a results-oriented culture, where leaders and staff debate questions that help them find, sustain, and spread promising practices and policies. 87

88 Pre and Post Decision Cross Cutting Tools 88

89 Tools and Techniques Cost Accumulation Cost Allocation Activity Based Costing Sensitivity Analysis 89

90 Tracing Costs from Expenditures to Products Cost accumulated by responsibility center A component of a reporting unit or organization Responsible for carrying out a mission, conducting a major activity, producing a product or providing services Cost allocated to cost objects The outputs of a responsibility center Indirect costs are spread over multiple Cost Objects. 90

91 Accumulation of Expenditures Job Order Costs are recorded for each unique job, product, batch or service separately Used for products, projects or assignments that differ in duration, complexity or resource requirements Process Cost are recorded and accumulated by process (one or more activities) Used for continuous flow production of homogeneous units undergoing identical procedures 91

92 Allocation of Expenditures Direct Tracing Observation, counting and recording of resources consumed by cost objects (outputs) Cause-and-Effect relationship Most accurate Most expensive and difficult Relatively easy for direct costs Data must exist Very difficulty for indirect (overhead) costs 92

93 Allocation of Expenditure Single-step Allocation Costs allocated to units based on a single cost driver (usually volume, labor hours, machine hours or direct materials, square meters) Least accurate Often the allocation base (cost driver) has little relation to the consumption of resources Least expensive and easiest 93

94 Tracing of Expenditures to Outputs Activity Based Costing Traces cost of resources to activities using resource drivers, then cost of activities to products using activity drivers Minimizes cost distortion Requires more effort and expense 94

95 Allocating costs for Wind Based Energy The commander has requested a cost breakdown by function for her new wind based energy initiative which will provide electricity for maintenance depots on base. She believes organizations on base can do much of the work to construct and maintain the new power source on a reimbursable basis and wishes to increase the budgets of these organization accordingly in order to pay for the new work load. The base comptroller recommends using Activity Based Costing to establish budget requirements to pay for the additional workload. Planning Construction Maintenance Power Distribution 95

96 Allocating costs for Wind Based Energy What do you Do? Resources benefits electricity supplies gasoline rent Resource Drivers Activities Planning Construction Maintenance Power Distribution 96

97 Allocating costs for Wind Based Energy Who do you do it for? Resources benefits electricity supplies gasoline rent Resource Drivers Activities Planning Construction Maintenance Power Distribution Activity Drivers Cost Objects Wind Energy Initiative Other work 97

98 Model Types of Uncertainty How we chose to build, structure, assess Subjective assessment Measurement Ability to assess/measure attributes Recording/transmission/transcribing errors Environmental Unforeseen future conditions 98

99 Sensitivity vs. Robustness Sensitivity analysis: explores how variation in parameters affects which alternative is the best; choice of alternative is sensitive if small or likely changes in parameter values change the choice of best alternative Robustness: an alternative is robust if it is good (i.e. effective and/or cost-effective) under many or most possible or likely future conditions 99

100 Assessment Techniques One-way sensitivity analyses Range of variance (BOP) Best, worst, most likely Analysis at each level Two-way analyses Multivariate analysis Simulation techniques 100

101 Example Suppose that the commander has three different options for weatherizing the buildings on base. Alternative A Hire an outside contractor Alternative B Have the work done by existing maintenance workers on base Alternative C Combination of contract labor and existing forces 101

102 Example (cont) In conducting a cost estimation, total costs have been shown to depend on a variety of cost components including Contracting costs Management costs Labor costs Materials costs Equipment costs 102

103 Assessment Techniques One-Way sensitivity analysis Where on the line is the breakpoint? Alternative A. Best Alternative B. Best Contracting Costs Breakpoint 103

104 Tornado Diagrams Comparison of one-way sensitivity analyses Let each component range from worst to best, all others held constant. Keep track of total value. Which attributes contributes most to variance? Total Cost Labor cost Materials cost Contracting cost Equipment cost Management Cost Baseline 104

105 Equipment Cost Assessment Techniques Two-Way sensitivity analysis Where in the graph are the changes? Alt. A is Best Alt. B is Best Alt. C is Best Labor Cost 105

106 Scenario Analysis Variation in the components is dependent on the scenario (operational conditions) Developing future alternatives Uncertain future conditions Looking for robustness across different scenarios 106

107 Important Questions Answered by Sensitivity Analysis Where should you spend more time and effort reducing uncertainty How much variation can happen before we change our choice? Which variables are most important to your decisions? What are the drivers? Are there attributes you may consider dropping? (because they don t affect choice) 107

108 Learning Objectives 1. Understand concepts underlying good decision analysis 2. Appreciate pre-decision analysis tools a. Cost-estimating b. Measuring effectiveness 3. Appreciate post-decision analysis tools a. Cost accounting b. Performance management 4. Specify, collect and use data 5. Be familiar with pre and post decision cross cutting accounting tools 6. Be familiar with sensitivity analyses and regression 108