The MACBETH approach to multicriteria value analysis

Transcription

1 The MACBETH approach to multicriteria value analysis Introduction of key issues to building a good MCA model by Carlos A. Bana e Costa Decision Support Models 2012/2013 The spirit of decision analysis is divide and conquer: decompose a complex problem into simpler problems, get one s thinking straight on these simpler problems, paste these analyses together with logical glue, and come out with a program of action for the complex problem Howard Raiffa 1968 Source: Raiffa, H. (1968). Decision Analysis: Introductory Lectures on Choices Under Uncertainty New York: Random House (p. 271) 1

2 Multicriteria Analysis Three fundamental advantages: Flexible and comprehensive, allowing for an easy integration of different interests and perspectives about the problem Allows for the aggregation, into the same model, of all sort of concerns (without the requirement to monetize them as in CBA) Allows taking into account objective and subjective elements in structuring, scoring and weighting costs, benefits and risks of options However Easily subject to misleading applications Several MCA methods do not have a sound theoretical basis The practice of Decision Analysis needs a sound theoretical basis (in the same way that we rely so firmly upon the natural sciences for our technological advances.) Elliot Jaques Source: Jaques, E. (1988). Requisite Organization New York: Random House (p. 271) to avoid serious mistakes in building value models and evaluating options 2

3 That basis is Decision Theory 3

4 M A C B E T H EASURING TTRACTIVENESS BY A ATEGORICAL ASED VALUATION EC NIQUE The MACBETH approach Purpose: To help people make better decisions Broad methodological framework: Decision Analysis (and Decision Conference) Type of multicriteria approach: Multi-criteria value measurement Specific type of modeling: Building quantitative value models based on qualitative value judgments Software: M-MACBETH decision support system 4

5 Phases of the MACBETH decision aiding process Analysis of the decision context and design of the intervention process structuring the problem MACBETH Sensitivity and robustness analyses and elaboration of recommendations Building the evaluation model 9 Decision Makers Fragmented options Context Stakeholders Types of Actors Experts Types of Options Alternatives Mixed options Facilitator Society Choice Context Types of Problems MACBETH Prioritisation Assignement 10 5

6 Example-case context: Evaluation of alternative new highway corridor. The Region and the three natural areas impacted want to use the same evaluation model Context MACBETH Direct evaluation of options Which type of evaluation model? Indirect evaluation of options Mixed model 6

7 AB4 WHICH EVALUATION FRAMEWORK to measure the attractiveness (desirability) of options on each value dimension? Performances Characteristics DESCRIPTOR (e.g. Indicator ) VALUE FUNCTION Allow to convert performance into value VALUE SCORES Characteristics PREFERENCE SCALES VALUE SCORES Objectives Structuring Neutral Good Structuring Actors 7

8 Slide 13 AB4 Where has the direct rating disappeared to? Andrea Brambilla;

9 Evaluation Actors Evaluation model Weighting Neutral Good Value Functions (for scoring) Set of Alternatives Recommendations Alt 4 Table of impacts Actors Recommendations 16 8

10 Set of Alternatives Recommendations Alt 4 Table of impacts Actors Recommendations Partial value scoring 17 Set of Alternatives Recommendations Alt 4 Table of impacts Actors Recommendations Partial value scoring Table of scores 18 9

11 Simple additive value model Once value functions and weights are established, the overall attractiveness of an option can be computed by the weighted sum of the partial value scores of its impacts on the criteria (j=1,,n=4): V(a) n j 1 k j.v j (I j (a)) Hierarchical value model Simple additive value model applied separately on each area of concern 10

12 Recommendations: Top-level dominance analysis Non dominated option Dominated options Recommendations: Sensitivity analysis profile of differences Sensitivity analysis on weight 22 11

13 Recommendations: Robustness analysis a) Uncertain performances b) Ordinal weighting a) and b) simultaneously 23 M MACBETH Decision Support System 12

14 M-MACBETH User s Guide 25 Common critical mistakes In Structuring Phase: An indicator is not a criterion. Means are not ends; causes are not effects. Redundancy of criteria gives rise to non requisite models. Scarce performance data does not necessarily implies that the respective criterion should not be considered in the analysis. In Evaluation Phase: Performance is not value. Subjectivity is not the same as arbitrariness. Weighting criteria based only on the notion of importance is the most common critical mistake. To rank is not to measure differences in value: To judge A better than B says nothing about how much A is better than B. Sum up ordinal scores on the criteria gives rise to meaningless overall scores. The method is not the decision maker; the model outputs are not unquestionable. THE MODELISNOTAPANACEA Neglect socio technical process design 13

15 Common critical mistakes In Structuring Phase: An indicator is not a criterion. Means are not ends; causes are not effects. Redundancy of criteria gives rise to non requisite models. Scarce performance data does not necessarily implies that the respective criterion should not be considered in the analysis. In Evaluation Phase: Performance is not value. Subjectivity is not the same as arbitrariness. Weighting criteria based only on the notion of importance is the most common critical mistake. To rank is not to measure differences in value: To judge A better than B says nothing about how much A is better than B. Sum up ordinal scores on the criteria gives rise to meaningless overall scores. The method is not the decision maker; the model outputs are not unquestionable. THE MODEL IS NOT A PANACEA Neglect socio technical process design Scoring options Ordinal vs. Cardinal, relative vs. intrinsic evaluation Example: Suppose there are three options B1, B2 and B3 Options should be ranked according to the relative attractiveness of their performances on each criterion B2 preferred to B1 preferred to B3 Is ranking by relative attractiveness enough to know if the most attractive option is good or bad? Defining Good and Neutral references resolves the issue B2 Good B1 B3 Neutral On each criterion, one wants to know not only if one option is more attractive than another but also by how much B2 Good B1 B3 Neutral weak Moderate 14

16 Assessing value judgments: MACBETH MACBETH is a interactive multicriteria decision support approach for:. Evaluate options on multiple criteria that uses qualitative judgments of differences in attractiveness in order to generate value scores for the options on each criterion and weights to the criteria. MACBETH introduces seven qualitative categories of difference in attractiveness: (Judgmental disagreement or hesitation between two or more consecutive categories, except indifference, is also allowed.) MACBETH process of scoring options on each criterion Step 1: Define Good and Neutral reference performances Step 2: Rank order options and references B2 Good B1 B3 Neutral Step 3: Use the MACBETH categories of difference of attractiveness to: Step 3.1: Evaluate qualitatively the difference between good and neutral Step 3.2: Evaluate qualitatively the difference between each option and each reference Step 3.3: Evaluate qualitatively the difference between each two options 15

17 Dealing with inconsistency Each time that a qualitative judgments is elicited, the consistency of all the judgments thereto made by the respondent is verified and suggestions are offered to resolve inconsistencies if they arise. Deriving scores From the consistent set of judgments. MACBETH derives a score for each option which the respondent should subsequently validate and may adjust if necessary. within a range compatible with the judgments elicited. 16