MACBETH Method, software and applications

Transcription

1 MACBETH Method, software and applications DECISION SUPPORT MODELS 2012/2013 The MACBETH Approach for Multi-criteria Decision Analysis Purpose: To help people make better decisions Broad methodological framework: Decision Analysis and Decision Conference Type of multicriteria approach: Multi-criteria value measurement Specific type of modeling: Building quantitative value models based on qualitative value judgments Software: M-MACBETH decision support system 1

2 Agenda Common critical mistakes The MACBETH method The M-MACBETH software MACBETH applications Behind M-MACBETH Common critical mistakes The method is not the decision-maker; the model outputs are not unquestionable. An indicator is not a criterion Means are not ends; causes are not effects Redundancy of criteria gives rise to non-requisite models Scarce performance data does not necessarily implies that the respective criterion should not be considered in the analysis Performance is not value Subjectivity is not the same as arbitrariness Weighting criteria based only on the notion of importance is the most common critical mistake To rank is not to measure differences in value; to judge A better than B says nothing about how much A is better than B sum up ordinal scores on the criteria gives rise to meaningless overall scores 2

3 Relative evaluation, absolute evaluation or both? Ordinal or cardinal evaluation? Example: Evaluation of the projects B1, B2 and B3 Relative evaluation: To choose the best option or to prioritize options Comparing the option one with each other B2 prefered to B1 prefered to B3 Absolute evaluation: Which options are good enough? Comparing eahc options with references of intrinsic value B2 Good B1 B3 Neutral Cardinal evaluation: How much is an option better than another? How good is an option? Evaluating diferences of atractiveness B2 Good B1 B3 Neutral Weak or moderate Moderate MACBETH MACBETH is a interactive multicriteria decision support approach for: Evaluate options on multiple criteria using qualitative judgments of differences in attractiveness in order to generate value scores for the options on each criterion and weights to the criteria. MACBETH introduces seven qualitative categories of difference in attractiveness: (Judgmental disagreement or hesitation between two or more consecutive categories, except indifference, is also allowed.) 3

4 MACBETH questioning procedure Pairwise comparison of options evaluating qualitatively their difference in attractiveness by selecting one MACBETH category (ex.: 'moderate difference between B1 and B3) or a sequence of categories in case of hesitation or disagreement (ex.: weak or moderate difference between B2 and Neutral) Each time that a qualitative judgment is elicited, the consistency of all the judgments made is verified Resolving inconsistencies In case of inconsistency... suggestions are offered to revise judgments to resolve the inconsistency 4

5 MACBETH process of scoring options on each criterion Step 1: Define Good and Neutral reference performances Step 2: Rank-order options and references B2 Good B1 B3 Neutral Step 3: Use the MACBETH categories of difference of attractiveness to: Step 3.1: Evaluate qualitatively the difference between good and neutral Step 3.2: Evaluate qualitatively the difference between each option and each reference Step 3.3: Evaluate qualitatively the difference between each two options 9 The scoring scale From the consistent set of judgments MACBETH derives a score for each bid which the respondent should subsequently validate and may adjust if necessary. within a range compatible with the judgments elicited. 5

6 MACBETH Procedure: Weighting Criterion j Good j Define on each criterion two reference levels of intrinsic value, that operationalize the idea of a good option Neutral j and a neutral option (that is, neither attractive nor unattractive) 11 Identifying good and neutral reference performances allows to launch the discussion on trade-offs in a natural and substantively meaningful way Benefit Risk Good B [B] [R] Good R Which of the two options (if any) is more attractive? Neutral B Neutral R 12 6

7 Always stay in touch with the current reality Quality Best Q a1 a2 Worst Q Quality Good Q a1 a2 Neutral Q Cost Best C Worst C Cost Good C Neutral C 1 st procedure Additional drawback: anchoring on best and worst can give rise to difficult or even unrealistic comparisons Proposed procedure Additional advantage: when anchoring on good and neutral, weights do not depend on options, but still depend on the decision context 13 Quality Cost Best C (very low cost) (best C, worst Q ) more attractive than (worst C, cost Q ) Good Q Best Q Worst Q Neutral Q Good C Neutral C Worst Q (very high cost) weight C > weight Q (neutral C, good Q ) more attractive than (good C, neutral Q ) weight C < weight Q Experience has shown that: Weights assigned to good-neutral ranges are close to intuitive weights 14 7

8 MACBETH Weighting Reference levels Good and Neutral set up the anchor points for meaningful weighting ) Evaluating the overall attractiveness of swinging from neutral to good on each criterion 16 8

9 MACBETH qualitative swing weighting procedure 1. Rank the good - neutral swings by their overall attractiveness 2. Judge qualitatively the overall attractiveness of each swing 17 MACBETH qualitative swing weighting procedure (cont.) 2. Compare qualitatively the most important swing to each of the others 3. Compare qualitatively each two swings consecutive in the ranking 4. If desired, complete the matrix of weighting judgements 18 9

10 Paired comparison of reference (hypothetical) options 19 From qualitative to quantitative weighting 5. Discuss and adjust the MACBETH weighting scale 20 10

11 The M-MACBETH Decision Support System 11

12 Phases of MACBETH decision aiding process Context analysis, problem framing and struturing the intervention Estructuring the evaluation elements M-MACBETH Sensitivity and robustness analyses Constructing the multicriteria evalaution model MACBETH Applications Evaluation of bids in public calls for tenders Evaluation of projects, policies and strategies Prioritization of investment projects Resource allocation Risk analysis Performance evaluation Location of infrastructures 12

13 Decision conference Development of socio-technical process of evaluation Evaluation and group learning; interactivity The facilitator leads the process without interfering in the content Proposed structuring framework: use MACBETH to combine direct with indirect evaluation approaches in the construction of descriptors based on reference scenarios 13

14 Economi c Overall value Social Ecological No (natural or proxy) descriptor of effects ENDS othe r Resilienc e MEANS Area Geometry Define Scenario 2 Scenario1 Baseline (no action) which improvement is VALUE SCALE Constructed descriptor of reference profiles of impacts on means Constructed descriptor of reference scenarios and respective equal-spaced value scale 14

15 Behind M-MACBETH Courtesy of João Bana e Costa (BANA Consulting) Behind M-MACBETH A B C D A Very Weak Moderate Very B Moderate Extreme 6 Very 5 4 C D Moderate 3 Weak 2 Very Weak 1 15

16 Behind M-MACBETH A B C D A Very Weak Moderate Very 1 B Moderate 3 C 4 D Extreme 6 Very 5 4 Moderate 3 Weak 2 Very Weak 1 Behind M-MACBETH A B C D A Very Weak Moderate Very 1 4 B Moderate 3 C 4 D Extreme 6 Very 5 4 Moderate 3 Weak 2 Very Weak 1 v(a)-v(c) = v(a)-v(b) + v(b)-v(c) 16

17 Behind M-MACBETH A B C D A Very Weak Moderate Very 1 4 B Moderate 3 C 4 D Extreme 6 Very 5 4 Moderate 3 Weak 2 Very Weak 1 v(a)-v(c) < v(c)-v(d) Behind M-MACBETH A B C D A Very Weak Moderate Very 1 4 B Moderate 3 C 5 D Extreme 7 Very 6 5 Moderate 3-4 Weak 2 Very Weak 1 Increase v(c)-v(d) of 1 17

18 Behind M-MACBETH A B C D A Very Weak Moderate Very 1 4 B Moderate 3 8 C 5 D Extreme 10 Very Moderate 3-4 Weak 2 Very Weak 1 v(b)-v(d) = v(b)-v(c) + v(c)-v(d) Behind M-MACBETH A B C D A no Very Weak Moderate Very B no Moderate 3 8 C no 5 D no Extreme 10 Very Moderate 3-4 Weak 2 Very Weak 1 v(a)-v(d) = v(a)-v(b) + v(b)-v(c)+ + v(c)-v(d) 18

19 Articles Bana e Costa, C. A., De Corte, J.-M., Vansnick, J.-C. (2011) MACBETH (Measuring Attractiveness by a Categorical Based Evaluation Technique), In: J.J. Cochran (Ed.), Wiley Encyclopedia of Operations Research and Management Science, Vol. 4, Wiley, New York. Bana e Costa C.A., De Corte J.-M., Vansnick J.-C. (forthcoming) On the mathematical foundations of MACBETH, J. Figueira, S. Greco, M. Ehrgott (eds.) In Multiple Criteria Decision Analysis: The State of the Art Surveys, Springer, Book Series: International Series in Operations Research & Management Science (update of the same book published in 2005, vol. 76, pp ). Bana e Costa, C.A., De Corte, J.M., Vansnick, J.C. (2012). MACBETH. International Journal of Information Technology & Decision Making, 11(2),