An evaluation of preference weighting methods in multicriteria analysis: testing on sediment remediation alternatives in the Bergen Harbour

Transcription

1 Sediment and Society - Technical Brief An evaluation of preference weighting methods in multicriteria analysis: testing on sediment remediation alternatives in the Bergen Harbour David N. Barton, Norwegian Institute for Nature Research(NINA) Multi-criteria analysis (MCA) is being used in the Sediment and Society project as a tool for communicating impacts, as a tool for documenting differences in interests of stakeholder groups, and as a tool for ranking complex sediment remediation alternatives. Ranking of alternatives in MCA requires impacts to be (i) standardised, and then (ii) weighted against one another. This Technical Brief evaluates alternative approaches that were tested by a group of researchers in a pilot evaluation session of the MCA software DEFINITE. The testing of DEFINITE finds that while more theoretically sound, the EValues procedure offered in the software package was more cumbersome to implement than the pairwise weighting of the analytical hierarchy procedures (AHP). This lead the team to conclude that further use of DEFINITE in the project would use AHP as understanding and legitimacy of the multi-criteria were deemed most important to the project, and biases of AHP could be identified. Introduction Multi-criteria analysis (MCA) is a stepwise process for ranking decision alternatives with multiple and often complex impacts. Information used in the stepwise process (Figure 1) is often structured Figure 1. Steps and roles in a multi-criteria analysis using a software tool. The software tool is used to record the alternatives (Step 2), how impacts are measured (Step3), and the information from the impact assessment (Steps 4 and 5). In Sediment and Society we have evaluated the MCA software DEFINITE (Janssen and Herwijnen, 2007). DEFINITE was chosen because it offers alternative methods for standardising different impacts to a common metric (Step 6), for weighting the relative importance of each impact (Step 7), and different algorithms for aggregating the weighted impacts for each alternative (Step 8). In this Technical Brief we evaluate two alternative approaches available in DEFINITE to Steps 6 and 7. - Alternative A: Direct standardisation using value functions & pairwise weighting of criteria - Alternative B: Indirect standardisation and weighting using DEFINITE EValues procedure A short explanation of standardisation and criteria weighting are needed to understand the difference between the two alternative approaches. Alternative A: Standardisation using value functions & direct pairwise weighting of criteria Standardisation of value functions Multiple impacts of each sediment remediation alternative cannot be compared or weighted without each impact first being standardised to a common metric. Standardisation requires technical experts and/or stakeholders to specify how an impact measured along some unitary scale (e.g. % excedance of per person annual PCB7 intake) is standardised, typically to a normalised value between 0=no impact and 1=full impact (Figure 2). Often the valuation function is considered linear for lack of knowledge on the importance of each incremental unit increase in

2 the impact. In Sediment and Society we wish to evaluate whether knowledge of the value function is best determined by technical experts or stakeholders. The argument in favour of technical experts doing standardisation is that environmental models may give clues as to whether the importance of an impact shows thresholds or is non-linear. The argument for stakeholders determining the shape of value functions lies in the term value in that there is subjective judgement, or valuation, involved in determining the incremental importance of each unit of impact (on stakeholders). Whether experts or stakeholders, the MCA literature indicates that determining the shape of the valuation function directly is an intellectually challenging and informationally intensive task (Beinat, 1997). Figure 3 Proposed hierarchy of impact criteria for Bergen Port sediment remediation MCA literature (Beinat, 1997) suggests that asking stakeholders so many direct weighting questions can be both cognitively challenging, as well as tedious. For this reason an indirect approach to determining criteria weights was devised by Beinat (1997), called EValues. Alternative B: Indirect standardization and weighting using DEFINITE EValues procedure Figure 2 A linear valuation function standardising the impact measured in physical units (horisontal axis) to a normalised impact value between 0 and 1(vertical axis). Alternatives labeled in the graph have different impacts described along the valuation function. Pairwise weighting of criteria Once impacts have been standardised they are mathematically comparable; but they are not necessarily equally important for different stakeholders. One of the most complete ways of determining stakeholders preferences for alternatives is by asking them to give a relative weight to all possible pairwise combinations of impact criteria. Sediment remediation alternatives for Bergen Harbour were compared using 10 different impact criteria, for a total of 50 pairwise comparisons. This task is simplified by criteria being organised in a hierarchy (Figure 3), with only criteria on the same level of each branch being compared pairwise (for a total of 15 pairwise comparisons). The EValues procedure uses three steps to indirectly elicit jointly(figure 4) (i1) participants rough standardisation of each impact using value function regions, (2) their ranking of criteria and (3) their preferences for hypothetical alternative decisions which vary for only two criteria at a time ( holistic profiles ) (Beinat 1997). The EValues algorithm uses these three types of information to calculate a ranking of the alternatives. Participants are taken through the three steps using the EValues functionality of the DEFINITE software. The three steps can be refined by repeating them in consecutive rounds until the participant agrees with the ranking of the alternatives (Figure 3). Note that Beinat (1997) calls participants experts, Figure 4. Steps in the EValues procedure Source: Beinat (1997)

3 whereas we have made the distinction between technical experts and stakeholders, because we argue that they play different roles in the decision process. Note also that stakeholders include technical experts as well as local expert knowledge and other participants. Participants start by defining a value region within which they think the standardisation of impact is located. Using the ranking of weights and hypothetical alternatives, EValues produces the following output: (i) a calculated value function, and (ii) weights for the different effects /impacts (Figure 4). These weights are then used to rank the sediment remediation alternatives. Group composition Intuitive ranking of sediment remediation alternatives Alternative Figure 4 Output 1: of Standardization EValues procedure using value functions Source: Janssen & direct and pairwise Herwijnen, weighting 2007 of criteria The pilot evaluation group Four researchers a limnologist/hydrologist, an environmental contaminants modeler, and two environmental economists - from the Norwegian Institute for Water Research (NIVA) were trained in the use of DEFINITE and asked to evaluate the two alternative MCA approaches as applied to the case of sediment remediation in Bergen Harbour. Prior to the one day evaluation, the researchers had reviewed a summary impact assessment of 5 and individually filled out a questionnaire. The questionnaire asked them some background questions, asked them to rank the 5 alternatives directly unaided by MCA - based on their reading of the impact assessment document, and finally asked them 15 pairwise weighting questions. Figure 5 shows that individual unaided ranking of alternatives quite consistently pointed to two remediation alternatives as the most preferable: capping all sediment and capping and dredging with shoreline disposal. Participants were then asked to specify value functions/standardisation directly for each of the 10 impacts, input their pairwise weighting data to DEFINITE, compute the ranking of alternatives and compare them. An internal validity test of the MCA methods evaluates whether the same ranking is repeated when aided by MCA. Participants were then asked to go through the EValues procedure using DEFINITE, refining their ranking of alternatives. Ranking results For all participants the alternative preferred was using MCA with direct standardisation and individual pairwise weighting of criteria (alternative A) produced the same priority remediation alternative as defined intuitively in Figure 5. The use of EValues to redefine/refine weighting did not change the ranking of alternatives (not illustrated here). Rather than emphasising the consistency of ranking results in such a small pilot evaluation, we focus below on the comparison of the two alternative weighting methods in terms of cognitive burden and user friendliness. Comparison of the two methods Alternative A: Standardisation using value functions & direct pairwise weighting of criteria Standardisation Figure 5 Unaided, intuitive ranking of alternatives by individual researchers sediment remediation alternatives for the harbour, -Effect/impact scores versus standardised values. Some participants stated that the shape of the value function could be hard to define for lack of an intuitive interpretation of the standardised value (vertical axis). By interpreting the impact as a dose and the standardised value as a response, participants found it easier to give a physical interpretation to the shape of the curve using e.g. expert knowledge of environmental modeling.

4 This dose-response interpretation may however be at odds with the concept of standardised values being normalised or dimensionless. -Missing impacts. Participants were critical of impacts that were lacking from the impact assessment (e.g. methane emissions from dredging versus capping the sea bed not being included in greenhouse gas calculations; or the benefits of business and recreational uses of the shoreline disposal site area). Participants argued that they sometimes factored in these missing impacts in their evaluation of the shape of the value function. -Value function as an expression of preferences. Participants also observed that the shape of the value function would depend on which experts were undertaking the evaluation (i.e. the valuation is also a subjective expression of values). -Context dependence. Participants also pointed out that the incremental importance of an impact depends on contextual factors not defined in the impact assessment (e.g. what other shoreline development plans exist for the area of the shoreline disposal site). Direct pairwise weighting -Clear definition of roles. Participants expressed difficulty in conducting weighting without knowing their role as stakeholders in the Bergen Harbour decision (participants were not residents). -Decision context dependence of weighting. Some participants expressed that it would have been difficult to conduct pairwise weighting if they had not had knowledge of the alternatives and their impacts (i.e. their preferences for impacts do not exists prior to or independently from the decision alternatives). -Missing impacts. Again respondents tended to factor in missing impacts into the relative weighting of criteria (based observation made by the moderator when listening to discussions on weighting between the participants). Overall evaluation Participants found that the correspondence of the unaided ranking with their direct MCA ranking raised awareness. They expressed that a condition for such consistency was nevertheless some level of prior knowledge of the alternatives and their impacts. Overall they did not find direct pairwise weighting to be tedious or an intellectual burden, given the current number of criteria (10). To the contrary, some expressed that pairwise weighting gave them a complete and transparent map of their own preferences. Alternative 2: Indirect standardisation and weighting using DEFINITE EValues procedure Despite being familiar with steps in Alternative A, participants found EValues to be more difficult because it forces evaluation (i.e. by using an algorithm to infer weights). Participants expressed a view that the EValues algorithm was more of a black box than pairwise weighting. Participants found indirect valuation of hypothetical alternatives harder than pairwise weighting of criteria. A software technical limitation expressed by participants in EValues (Step 3 - the ranking of hypothetical alternatives holistic profiles ), was that alternatives could only be ranked as equal or worse than the neighbor alternative to which it was being compared, but never better than. Overall evaluation While participants found the EValues procedure to be more cumbersome than the direct pairwise weighting approach, EValues did expose a number of weaknesses or inconsistencies in the impact assessment document: Location cannot be characterised as an impact of itself, but must be further disaggregated into effects that can be characterised as costs or benefits depending on the location of the stakeholders affected - e.g. number og households affected by remediation works. Dynamic impacts/time. The impact analysis was prepared as a mixture of immediate impacts and dynamic impacts on PCB7 diffusion and health impacts. The dynamic impacts are non-linear and could have been described using separate impact matrices for a number of years into the future. Participants attempted to integrate dynamics into the value functions (see standardisation above). The correlation between impact criteria makes the generation of hypothetical alternatives in the EValues procedures unrealistic in many cases.

5 Conclusions Conclusions: Weighting approach The researchers in the training session on DEFINITE reached a consensus that Alternative A involving direct standardisation of value functions and direct pairwise weighting of impact criteria, was more transparent than Alternative B using the indirect and holistic approach of EValues procedure in DEFINITE. Two main reasons for the relative weakness of EValues to the direct approach were: - Some perceived technical weaknesses in the user interface of EValues - The algorithm for deriving impact standardisation and weights in EValues was perceived as more black box than direct pairwise weighting. The additional burden of having to weight all criteria pairs was not seen as problematic by participants. Conclusions: Division of roles in MCA The EValues procedure requires a participant to jointly standardise and weight impacts. Participants in our evaluation thought that standardisation of impacts should not be determined by stakeholders; determining the shape of a standardisation function requires some dose-response or technical knowledge which stakeholders may not have. While participants had no opinion on who should carry out weighting of impacts, theory suggests that it should be carried out by stakeholders, as it requires value judgements with consequences for other stakeholders. Our recommendation: Based on the evaluation of the DEFINITE software described above, we recommend that experts conduct standardisation of impacts, which can be reviewed and approved by stakeholders. This will capture technical expert knowledge about functional form, while allowing stakeholders to approve any valuation judgements made by technical experts. Stakeholders then carry out weighting. EValues may be potentially useful to analysts as a consistency check on the definition of impact criteria, before conducting pairwise weighting with stakeholders. References Beinat, E. (1997) Environmental Valuation Functions. Kluwer. Janssen, R. & Herwijnen, M. v. 2007, DEFINITE 3.1. A system to support decisions on a finite set of alternatives (Software package and user manual) Institute for Environmental Studies (IVM), Vrije Universiteit, Amsterdam