Evaluation Models. Multi-Attribute Models. Multi-Attribute Modelling: Why? Multi-Attribute Model Structure. Decision Support: Multi-Attribute Methods

Transcription

1 : : Multi-Attribute Methods, Ljubljana Programme: Information and Communication Technologies [ICT3] Course Web Page: Institut Jožef Stefan, Department of Knowledge Technologies, Ljubljana and University of Nova Gorica Evaluation Models alternatives EVALUATION EVALUATION MODEL ANALYSIS Multi-Attribute Models cars buying Multi-Attribute Modelling: Why? Systematic, structured approach (to difficult real-life problems) Model development: problem decomposition into smaller, less-complex subproblems requires understanding and careful elaboration of the problem facilitates and motivates communication and knowledge interchange maint safety doors TECH Analysis: what-if analysis sensitivity analysis explanation: how? (evaluation procedure) why? (selective explanation of advantages/disadvantages) pers lug COMF problem decomposition Contributes to better decisions: understanding, justification, explanation, documentation Multi-Attribute Model Structure Multi-Attribute Model for Car Selection Y (UTILITY) F(X 1,X 2,,X n ) FUNCTION F(X 1,X 2,,X n ) FUNCTION X 1 X 2 X n ATTRIBUTES FUEL SAFETY ATTRIBUTES a 1 a 2 a 3 a1 a 2 a 3 a 1 a 2 a n ALTERNATIVES a 1 a 2 a 3 a1 a 2 a 3 a 1 a 2 a n S Ljubljana, Slovenia 1

2 : Quantitative Multi-Attribute Model for Car Selection FUNCTION P P P 3 FUEL SAFETY AGGREGATION FUNCTION MARGINAL FUNCTIONS Qualitative Multi-Attribute Model for Car Selection FUNCTION (DECISION RULES) high high unacc unacc low low unacc unacc high low unacc med high unacc med med acc acc med med acc low low acc acc low med med low low low FUEL SAFETY med high acc 2. high low 3. low med acc unacc unacc acc ALTERNATIVES ALTERNATIVES Hierarchical Multi-Attribute Model Multi-Attribute Modelling: How? Y F(X6,X4,X5) X6 F(X1,X2,X3) X1 X2 X3 X4 X5 utility value (utility) utility value function function aggregate aggregate attributes attributes utility value function function basic basic attributes attributes 0. Problem identification 1. Tree (or hierarchy) of attributes 2. Utility functions 3. Evaluation and analysis of alternatives 4+ Implementation a1 a2 a3 a4 a5 alternative (option) alternative 1. Tree of Attributes Exercise Decomposition of the problem to to sub-problems ("Divide and Conquer!") You would like to buy a new laptop computer for your own purposes (study, internet, fun,...). TECH.CHAR. Suggest a suitable set of attributes and create a tree of attributes. BUYING MAINTEN SAFETY COMFORT Consider the guidelines presented on the next two slides. The most difficult stage! Ljubljana, Slovenia 2

3 : Developing Attribute Structure Three basic strategies: Top-Down: Start with the overall evaluation (target objective), decompose it to sub-goals. Bottom-Up: Start with desirable characteristics, subgoals. Group them into connected, meaningful sub-trees. Middle-Out: Combining the two above. Iteratively decompose (refine) and group (generalise) attributes. Developing Attribute Structure Desirable features of attributes and their structure: Completeness: Do not overlook important attributes Relevance (non-redundancy): Use only relevant attributes, omit redundant attributes Minimality: Use a minimal number of attributes Orthogonality: Basic attributes should be independent of each other Operativity: Basic attributes should be easy to assess or measure Comprehensibility: Create meaningful sub-trees of interrelated attributes 2. Utility Functions (Aggregation) 3. Evaluation and Analysis Aggregation: bottom-up aggregation of attributes values TECH.CHAR. EVALUATION buying maint direction: bottom-up (terminal root attributes) result: each alternative evaluated inacurate/uncertain data? 75% 25% safety BUYING MAINTEN SAFETY COMFORT doors TECH pers COMF SAFETY COMFORT TECH.CH. lug low exc unacc high low unacc med accept accept high exc 3. Evaluation and Analysis MADM Tools ANALYSIS buying maint interactive inspection what-if analysis sensitivity analysis explanation 1. Paper and Pencil (Abacon) 2. Spreadsheets and mathematical modelling software(ms Excel) 3. Specialized MADM software safety doors TECH pers COMF lug Ljubljana, Slovenia 3

4 : Spreadsheet Modelling Specialized Software (1/5) Logical Decisions Criterium DecisionPlus WinPre Downloadables/winpre.html Specialized Software (2/5) DECERNS Specialized Software (3/5) Web-HIPRE HiView Specialized Software (4/5) D-SIGHT Specialized Software (5/5) DEXi Ljubljana, Slovenia 4

5 : Working Example Kepner-Tregoe One Thursday morning, Charles, instead of attending his Management Science Techniques for Consultants class, was mulling over his four job offers. His offers came from: Acme Manufacturing, Bankers Bank, Creative Consulting, and Dynamic Decision Making. He knew that factors such as,, amount of management science (which he loved), and long term prospects were important to him, but he wanted some way to formalize the relative importance, and some way to evaluate each job offer. Kepner, C. H., Tregoe, B. B. (1981). The New Rational Manager. Princeton Research Press. Characteristics: list of attributes importance of attributes is expressed by weights [0,10] alternatives are described by vectors of values [0,10] evaluation (aggregation) principle: weighted sum supported analyses: what-if, sensitivity Adapted from: Michael A. Trick, Analytic Hierarchy Process, Kepner-Tregoe Model Kepner-Tregoe: What-If Analysis Job Offers Method: Kepner-Tregoe alternative A B C D attribute weight value w*v value w*v value w*v value w*v management science long-term prospects total Job Offers Method: Kepner-Tregoe alternative C original C - C2 attribute weight value w*v value w*v value w*v management science long-term prospects total Kepner-Tregoe Model: Charts Kepner-Tregoe: Sensitivity Analysis long-term prospects Weights long-term prospects; 3 Weights ; 2 Sensitivity analysis Attributes management science management science; Va lues long-term prospects Attribute Values ; 10 Value A B C D Value Attribute management science Weight of 0 A B V alue Stanovanje A B C D Ljubljana, Slovenia 5

6 : AHP Hierarchy of Attributes AHP: Analytic Hierarchy Process (Thomas Saaty, 1980) JOB WEIGHTS Characteristics: based on multiple attribute hierarchies assessing weights by a pairwise comparison of attributes assessing preferences by a pairwise comparison of alternatives consistency analysis (BASIC) ATTRIBUTES MS long PREFERENCES ALTERNATIVES Acme Bankers Creative Dynamic Pairwise Comparison Values 1 Items i and j are of equal importance (preference) 3 Item i is weakly more important (better) than j 5 Item i is strongly more important (better) than j 7 Item i is very strongly more important (better) than j 9 Item i is absolutely more important (better) than j 2,4,6,8 are intermediate values Assessing Weights Location Salary MS Long Location 1 1/5 1/3 1/2 Salary MS 3 1/2 1 3 Long 2 1/4 1/ Normalize the columns so that the sum equals 1 2. Take the average of rows. Location Salary MS Long Weights Location Salary MS Long Assessing Preferences (Scores) For each attribute, e.g., Location, compare alternatives: Location A B C D A 1 1/2 1/3 5 B 2 1 1/2 7 C D 1/5 1/7 1/ Normalize the columns so that the sum equals 1 2. Take the average of rows. Location A B C D Preferences A B C D Assessing Preferences (Scores) Scores for all the attributes: A B C D Location Salary MS Long Evaluation: [WSM] = ( ) Acme: (.086)(.174)+(.496)(.050)+(.289)(.210)+(.130)(.510)=0.164 Banks: = Creative: = Dynamic: = Ljubljana, Slovenia 6

7 : AHP Software Criterium DecisionPlus DECERNS Web-HIPRE Software Web-HIPRE WinPre Downloadables/winpre.html Multicriteria Modeling Software DEX (Decision EXpert): Program Method URL 1000Minds PAPRIKA Criterium DecisionPlus AHP, SMART Decision Deck MCDA Decision Lab PROMETHEE & GAIA MCDA DecisionPad DEXi DEX D-SIGHT PROMETHEE Expert Choice AHP GMAA MCDA HIPRE AHP, SMART Hiview MAUT Logical Decisions MCDA MakeItRational AHP M-MACBETH MACBETH V.I.S.A MCDA Visual PROMETHEE PROMETHEE & GAIA UTA Visual UTA Web-HIPRE SMART..AHP Winpre AHP Qualitative Multi-Attribute Modelling Method DEXi: DEX for Education Computer Program for Multi-Attribute Decision Making DEX (Decision EXpert) Method for qualitative multi-attribute modeling DEX is similar to other multi-attribute methods: 1. Multiple attributes, hierarchically structured 2. Evaluation of alternatives: bottom-up aggregation DEX Method for qualitative multi-attribute modeling DEX is different from other multi-attribute methods: 1. Attributes are discrete, symbolic, qualitative TECH.CH. TECH.CH. BUYING MAINT FUEL SAFETY COMFORT BUYING MAINT FUEL SAFETY COMFORT Some Car Numeric (quantitative): BUYING = numeric scale, e.g. R + Symbolic (qualitative): BUYING = medium scale: {high, medium, low} Ljubljana, Slovenia 7

8 : DEX Method for qualitative multi-attribute modeling DEX is different from other multi-attribute methods: 2. Evaluation of alternatives (aggregation) is defined by decision rules DEX Method for qualitative multi-attribute modeling DEX is different from other multi-attribute methods: 2. Evaluation of alternatives (aggregation) is defined by decision rules = 75%*BUYING + 25%*MAINT = 75%*BUYING + 25%*MAINT BUYING 75% 25% MAINT FUEL TECH.CH. SAFETY COMFORT BUYING 75% 25% MAINT TECH.CH. FUEL SAFETY COMFORT TECH.CH. high exc unacc low bad med unacc med med FUEL SAFETY COMFORT DEX Method: History Methodology initial development Software DECMAK toolbag First applications HW and SW selection personnel mgmt nursery schools DECMAK Methodology integration Software DEX Vredana National applications Housing Fund Ministry Sci-Tech Talent System industry medicine Related HINT DEX Methodology further improvement Software DEXi Education International applications Sol-Eu-Net agriculture, food, GMO project evaluation finance Related model revision, prodex DEXi DEXi Computer Program for Multi-Attribute Decision Making A simple computer program for MADM that facilitates: Creation and editing of model structure (tree of attributes) value scales of attributes decision rules (incl. using weights) alternatives and their descriptions (data) Evaluation of alternatives (can handle missing values) Presentation of evaluation results with: tables charts Analyses: what-if, ±1, selective explanation, comparison Preparing reports and charts DEXi Model Attribute Scale Job unacc; acc; ; exc unacc; acc; unacc; acc; satisfaction unacc; acc; MS unacc; acc; long unacc; acc; Tables satisfaction Job 33% 33% 33% 1 unacc * * unacc 2 * unacc * unacc 3 * * unacc unacc 4 acc acc >=acc acc 5 acc >=acc acc acc 6 >=acc acc acc acc 7 acc 8 acc 9 acc 10 exc DEXi Evaluation Evaluation results Attribute A B C D Job unacc unacc acc unacc acc acc unacc unacc acc acc satisfaction unacc acc MS acc unacc acc long unacc acc acc acc satisfaction B D acc satisfaction MS long satisfaction 50% 50% 1 unacc * unacc 2 * unacc unacc 3 acc acc acc 4 >=acc 5 >=acc acc acc satisfaction acc satisfaction Ljubljana, Slovenia 8

9 : Stages of MADM with DEXi 0. Problem Identification a. problem formulation b. formation of a decision-making group c. selection of decision-support methodology 1. Identification of Attributes a. unstructured list of attributes b. hierarchy (tree) of attributes c. measurement scales 2. Definition of Utility Functions (Decision Rules) 3. Evaluation and Analysis of Alternatives a. description of alternatives (data acquisition) b. evaluation of alternatives c. analysis 4. Implementation Stages of MADM (with DEXi) 0. Problem Identification a. problem formulation b. formation of a decision-making group c. selection of decision-support methodology DECOMPOSITION AGGREGATION 1. Identification of Attributes a. unstructured list of attributes b. hierarchy (tree) of attributes knowledge acquisition c. measurement scales 2. Definition of Utility Functions (Decision Rules) 3. Evaluation and Analysis of Alternatives a. description of alternatives (data acquisition) b. evaluation of alternatives c. analysis 4. Implementation EXPLOITATION 1.a: Unstructured List of Attributes Problem in Personnel Management: Select of a Candidate for a Job (e.g., a project manager) education age experience references knowledge work approach ability to work in a group Do not overlook important attributes! leadership organizational abilities loyalty intelligence communicativity character health 1.b: Tree of Attributes 1.b: Tree of Attributes Employ Educat Years Personal Formal For.Lang Exper Age Abilit Test Comm Leader Create meaningful, related groups Avoid aggregate attributes having more than three descendants Ljubljana, Slovenia 9

10 : 1.c: Scales 1.c: Scales Employ unacc, acc,, exc unacc, acc, Educat Years Personal unacc, acc, Formal For.Lang Exper Age Abilit prim/sec, high, univ, MSc, PhD no, pas, act none, to1year, 1-5, 6-8, more 18-20, 21-25, 26-40, 41-55, more Comm poor, aver,, exc unacc, acc, Leader Test D, C, B, A less, approp, more Scales are discrete, typically ordered from bad to Values should distinguish between importantly different characteristics Their number should gradually increase from bottom to the root 2: Decision rules Utility Functions, Bottom-Up Aggregation Comm Leader Abilit poor less unacc poor approp unacc poor more unacc aver less unacc aver approp acc aver more acc less unacc approp acc more exc less unacc exc approp exc more Personal Abilit Comm Leader Test 2: Decision rules 3.a: Description of Alternatives 3.a: Description of Alternatives Employ Educat Years Personal Formal For.Lang Exper Age Abilit Test Comm Leader Candidate Formal For.Lang Exper Age Comm Leader Test A MSc pas to1year more B B PhD act more aver less B Ljubljana, Slovenia 10

11 : 3.bc: Evaluation/Analysis of Alternatives 3.b: Evaluation of an Alternative 1. Evaluation proceeds from bottom (basic attributes) to the root result: qualitative evaluation of each alternative handles missing (DEXi) or imprecise (DEX) alternative values Candidate A Employ Educat acc Years acc Personal 2. Analysis interactive inspection of results what-if analysis analyses: compare alternatives ±1 analysis selective explanation reports charts Formal For.Lang Exper Age Abilit Test MSc pas to1year B Comm Leader more 3.b: Evaluation of Alternatives 3.c: What-If Analysis Candidate A Employ no effect Educat acc Years acc Personal Formal For.Lang Exper Age Abilit MSc PhD pas to1year B Comm Leader more Test 3.c: What-If Analysis 3.c: What-If Analysis Candidate A Employ exc acc Educat Years acc Personal Formal For.Lang Exper Age Abilit MSc Test pas to1year B act Comm Leader more Ljubljana, Slovenia 11

12 : 3.c: ±1 Analysis 3.c: Compare Alternatives 3.c: Selective Explanation 3.c: Selective Explanation Candidate B Employ unacc Educat Years Personal unacc unacc Formal For.Lang Exper Age Abilit Test PhD act more B Comm aver Leader less Charts and Reports DEX and DEXi: Experience Wide applicability to various application areas Usually, solutions are specific (non-general) 1. Model development time heavily problem-dependent: from hours to months typical: 2 to 15 days 2. The most difficult stage designing the tree of attributes 3. Appropriate decision problems many attributes (> 15) many alternatives (> 10) prevailing qualitative decision-making, judgment inaccurate or missing data group decision making (communication and explanation) sufficient resources available (expertise, time) Ljubljana, Slovenia 12

13 : DEX and DEXi: Summary 1. Combination of multi-attribute decision making and expert systems 2. Characteristics: qualitative (symbolic) decision making explanation and analysis active support in the acquisition of decision rules 3. Applicability: for complex real-world problems hundreds of real-life applications Ljubljana, Slovenia 13