QUANTIFYING HAZARDS AND RISKS WITH EXPERT JUDGMENT. Willy Aspinall

Transcription

1 4 6 November 2009 QUANTIFYING HAZARDS AND RISKS WITH EXPERT JUDGMENT Willy Aspinall Bristol University / Aspinall & Associates Willy.Aspinall@Bristol.ac.uk

2 Promises, promises... our work/project/research will reduce uncertainty

3 The Three Horsemen of Risk Apocalypse with apologies to Roger Cooke UNCERTAINTY AMBIGUITY INDECISION

4 The Three Horsemen - example UNCERTAINTY PF goes which way, how far and how fast? AMBIGUITY What is understood by term pyroclastic flow? INDECISION Do we evacuate?

5 The Three Horsemen - responses UNCERTAINTY Do measurements, quantify uncertainty AMBIGUITY Define concepts, domain of application INDECISION Assess utilities, preferences

6 The Three Horsemen - roles UNCERTAINTY Experts role to quantify AMBIGUITY Analyst/facilitator s job to clarify INDECISION Stakeholder, problem owner s responsibility

7 Elicitation of expert judgment In climate change modelling, for instance, the challenges are exemplified by:. We explore a high rate of refusal to participate in this expert survey: many scientists prefer to rely on output from future climate model simulations. Arnell, N. W., E. L. Tompkins, et al. (2005). Eliciting Information from Experts on the Likelihood of Rapid Climate Change. Risk Analysis 25: The past performance of such projections has been systematically overconfident. Analysts have often used scenarios based on detailed story lines. for evaluating uncertainty. No probabilities are typically assigned to such scenarios. Morgan, M.G. and D. Keith (2008). Improving the way we think about projecting future energy use and emissions of carbon dioxide. Climatic Change 90:

8 The Classical Model A performance-based procedure for quantifying uncertainties from expert judgments Q i = Pr(event data)??? W j = C j * I j Cooke, R.M. (1991) Experts in Uncertainty. OUP. Cooke, R.M. and L.L.H.J. Goossens (2008) TU Delft expert judgment data base. Reliability Engineering & System Safety Expert Judgement 93: Synthesised group Decision-Maker DM i = W j *Q i

9 One case history (of several) DEFRA study objective: to develop a generic quantitative model for accelerated internal erosion in Britain s population of 2,500 ageing dams, using elicited quantities for key variables Cowlyd Reservoir inspection party Warmwithens Dam failure risk assessment and reservoir safety in the UK

10 Experts spreads of opinion for one parameter Opinions on the time-to-failure (in days from first detection) for the 10%ile of slowest cases... and outcomes obtained by alternative ways of weighting and pooling opinions Note the two schools of thought effect and the strong opinionation of many experts

11 The reservoir engineers: performance-based scores, and mutual weighting rankings Calibration weights versus mutual weights

12 Equal weights, performance-based weights and an expert census approach hypothetical SSHAC-4 expert census uncertainty spread??

13 Advanced 3D computational fluid dynamics modelling Courtesy INGV and EU EXPLORIS Project

14 Elicitation of realistic physical uncertainties on model outputs

15 Analysing expert elicitations with Cooke s Classical Model The procedure relies on cornerstones of the scientific method: Empirical control - evaluates weights for experts on basis of measures of performance Accountability - inputs are traceable in terms of scientific inputs of individuals Reproducibility - can replicate and review all calculations used Advantages: Impartiality - experts are treated equally prior to calibration Equity individual experts scores are maximised by stating true scientific views Diagnostic - procedure can highlight discrepancies in reasoning or inconsistencies in interpretation this approach produces a rational consensus, and sits squarely within the Bayesian paradigm for decision-support

16 Montserrat - 11 October 2009

17 Probabilistic forecasting for Montserrat volcano using the structured expert elicitation approach 2. GIVEN current conditions, what is the probability that within the next year the first significant development will be the resumption of lava extrusion. Credible interval lower bound Median estimate Credible interval upper bound SAC elicitation 6.3% 34.1% 66.1%

18 Forecast metric - Brier Skill Score BS Brier Score 1 n n k 1 f k o k 2 o i = 1 if the event occurs = 0 if the event does not occur f i is the probability of occurrence according to the forecast system BS can take on values in the range [0,1], a perfect forecast having BS = 0 Brier Skill Score BSS BS cli BS BS cli o 1 cli BS o The forecast system has predictive skill relative to some reference (e.g. climate record) if BSS is positive, a perfect system having BSS = 1. = total frequency of the event o (e.g. sample climatology / global data / other reference basis)

19 Forecast skill performance of Montserrat SAC

20 Probabilistic forecast scorecard All forecasts (110 no.) Life critical forecasts (75 no.) +ve BSS 84 (76%) 61 (83%) zero or -ve BSS 26 (24%) 14* (17%) * includes some most threatening scenarios cautious

21 Cumulative Return on Investment ROI Sep-2008 Sep-2006 Sep-2004 Sep-2002 Sep-2000 Sep-1998 Sep-1996 Sep-1994 Communicating forecast skill Montserrat case, following Lenny Smith & colleagues Surrogate metrics for forecast skill ROI [1 staked per forecast] [Hagedorn, R., Smith, L.A. (2008) Communicating the value of probabilistic forecasts with weather roulette. Meteorol. Appl. Published online in Wiley InterScience ( DOI: /met.9. ]

22 Brier Score rel. uniform probs Forecast performance versus outlook period Brier Score by outlook period 1.5 Brier Score by outlook period Months

23 Challenging elicitations of scientific expert judgment The Harvard study on Kuwait s First Gulf War reparations claim More Than 700 Fires First Fires Air War ~ 17 January 1991 Ground War ~ 23 February 1991 Liberation ~ 28 February 1991 Last Fire - 6 November 1991 Oil Burned ~ 4 x 10 6 barrels per day PM Emissions ~ 3 x 10 9 kg PM10 levels typical 300 Health effects ug/m3, claim sometimes based 2000 on expert elicitation: ~ 35 deaths Individual experts best mortality estimates: 13, 32, 54, 110, 164, 2874 Equal Weights (82 deaths; 90% conf. range: 18 to 400 ) Performance Weights (35 deaths; 16 to 54) The judicial decision of the UN Commission eventually rejected the admissibility of this form of evidence: not actual data.. and we won t mention Prof Nutt and cannabis!

24 Estimating dose-response curves for cancer risk from airborne arsenic using expert inputs Work with the late Joey Hanzich (Env. Epid. MPhil ) and Dr Peter Baxter at IPH Cambridge

25 Weighted Cumulative Probability Extracting signal from expert noise Weighted Cumulative Probability vs Cumulative Exposure Example self-weighted curves from one individual expert for one risk ratio value Estimated Risk Ratio and pooled results for group, combined with EXCALIBUR weights Cumulative Exposure in (mg/cubic m)*years

26 Performance A supplementary approach The Cooke Classical Model and EXCALIBUR procedure for eliciting quantitative values and uncertainty distributions from multiple experts. For more qualitative assessments of uncertain factors, simple paired comparison analysis using Probabilistic Inversion (PI) model fitting provides an alternative way of characterizing relative rankings ( revealed preferences ) from a group, with quantitative estimates of associated uncertainties: Two-factor ranking of option items by Paired Comparison with Probabilistic Inversion Item 10 Item 4 Item 7 Item 9 Item 6 Item 3 Perform. Std. dev. Import. Std. dev. Item Item Item Item 1 Item 2 Item Item Item Item8 Item Item Item 5 Item Item Importance

27 In almost all circumstances, and at all times, we find ourselves in a state of uncertainty - Bruno de Finetti.and scientists will continue to be perplexed, bemused and uncertain!

28 Summing up... our work/project/research will reduce uncertainty. a laudable goal, but the opposite is likely to emerge when exhaustive and formalized investigations of scientific uncertainty are undertaken and scientists will have to think how best to communicate the implications for hazard and risk management! Thank you!