Sobol s sensitivity indices beyond sensitivity. Session in honour of Prof. Ilya M. Sobol' By Art Owen and Sergei Kucherenko

Transcription

1 Eleventh International Conference on Monte Carlo and Quasi-Monte Carlo Methods in Scientific Computing, Leuven, April 2014 Sobol s sensitivity indices beyond sensitivity Andrea Saltelli, William Becker, Paolo Paruolo & Michaela Saisana Unit of Econometrics and Applied Statistics, Joint Research Centre. European Commission Session in honour of Prof. Ilya M. Sobol' By Art Owen and Sergei Kucherenko

2 Working on Sobol measures since 1990 s ( ( )) V E Y X X X i i ~ i This is a first order effect, or top marginal variance. This is a total order effect, or bottom marginal variance. The expected variance than would be left if all factors but Xi could be fixed. The expected reduction in variance than would be achieved if factor Xi could be fixed. E ( ( )) V Y X X ~ X i i ~ i

3 Two papers with Prof. Sobol many years ago Sensitivity measures, ANOVA-like techniques and the use of bootstrap, 1997, GEB Archer, A Saltelli, IM Sobol, Journal of Statistical Computation and Simulation, 58 (2), About the use of rank transformation in sensitivity analysis of model output, 1995, A Saltelli, IM Sobol, Reliability Engineering & System Safety, 50 (3),

4 Today two non sensitivity applications 1) To composite indicators 2) To variable selection in regression

5 About indicators

6 Advocacy, analysis and quality Composite indicators are models. They live in a space whose main axes are advocacy, analysis and quality. Composite indicators sit between analysis and advocacy, but quality discriminates the plausible from the rhetorical.

7 Fortune of composite indicators Statistics' best known face (to general public & media) An example, about Côte d Ivoire, Economist October July 2013 One of the indicators in the ruling justly category is control of corruption, an area in which Côte d Ivoire fares particularly poorly. The World Bank s most recent corruption rankings, from 2011, put it 38th out of 49 African countries. Transparency International ranked it 130th out of 176 countries last year in its Corruption Perceptions Index, ahead of Nigeria and Guinea but well behind neighbouring Liberia, Burkina Faso and Ghana.

8 Quality of composite indicators

9 Sensitivity analysis, invasive and non-invasive Testing (composite) indicators: two approaches Michaela Saisana, Andrea. Saltelli, Stefano Tarantola (2005), Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators, J. R. Statist. Soc. A 168(2), Paolo Paruolo, Michaela Saisana, Andrea Saltelli, (2013), Ratings and rankings: Voodoo or Science?, J. R. Statist. Soc. A, 176 (3),

10 Non invasive Sensitivity analysis Comparing the weights as assigned by developers with effective weights derived from sensitivity analysis.

11 Non invasive Sensitivity analysis HDI 2009 declared weight importance Life expectancy, 33% Adult literacy, 22% Enrollment education, 11% GDP per capita, 33%

12 Non invasive Sensitivity analysis HDI 2010 declared weight importance Life expectancy, 33% Education, 33% GNI per capita, 33%

13 Non invasive Sensitivity analysis declared weight importance HDI 2010 more coherent than HDI 2009

14 Two applications of non invasive sensitivity analysis 14

15 How is this done? Pearson s correlation ratio Smoothed curve First order sensitivity index Unconditional variance

16 How is this done? Smoothed curve First order sensitivity index:

17 How is this done? The measure Si is the numerical translation of the shape of a scatterplot S i ~ 0.3 S i ~ 0.4 EPI 2012 Environmental Health (EH) Ecosystem Vitality (EV)

18 Selection of variables in regression A new approach to variable selection in regression using a sensitivity analysis measure

19 Selection of variables in regression Ever heard about I JUST RAN FOUR MILLION REGRESSIONS? Sala-i-Martin, X. (1997). I just ran two million regressions. The American Economic Review, Papers and Proceedings of the Hundred and Fourth Annual Meeting of the American Economic Association 87,

20 Selection of variables in regression Xavier X. Sala-i-Martin

21 Selection of variables in regression We started from Hoover and Perez K.D. Hoover and S.J. Perez. Data mining reconsidered: encompassing and the general-to-specific approach to specification search. The Econometrics Journal, 2(2): , When comparing optimized algorithms, the GSA method appears to be able to reduce the failure rate in recovering the EDGP from 5% to 1% approximately - a five-fold reduction. GSA global sensitivity analysis EDGP effective data generating process

22 The experiment in pills 1. There is a pool of 18 variables and two time lags=36 possible choices 2. Eleven models, or data generating processes (DGP s) are created using different variables for the pool 3. Using a given model/dgp s outputs and the full pool of variables one tries to discover the variables in the model/dgp by randomly selecting variables and computing a measure of fit betwee the trial regression and the DGP s known output.

23 The experiment in pills 4. This random selection for the trials can be done generating for each variable variable Xi in the pool a random numbers Zi. 5. If a random number Zi is less or equalo to 0.5 then Xi goes into the regression model, otherwise not. Zi is a trigger. 6. Compute the total sensitivity index of the measure of fit with respect to the triggers Zi over a set of trials on the DGP s 7. Select as final variables those with the highest total sensitivity index

24 More details: Exploring Hoover and Perez's experimental designs using global sensitivity analysis, by William Becker, Paolo Paruolo, and Andrea Saltelli Cite as: arxiv: [stat.co]

25 And now for something completely different

26 Sensitivity auditing seven rules Seven rules for the use of mathematical modelling at the science policy interface

27 RULE ONE: Check against rhetorical use of mathematical modelling Don t use models like Latin

28 RULE THREE: find sensitivities before sensitivities find you;

29 RULE FOUR: detect GIGO (Garbage In, Garbage Out) Check that uncertainty has not be inflated or deflated

30 RULE SEVEN: Explore diligently the space of the assumptions Don t move an assumption at a time

31 More reading Saltelli, A., Guimarães Pereira, Â., Van der Sluijs, J.P. and Funtowicz, S., 2013, What do I make of your latinorum? Sensitivity auditing of mathematical modelling, Int. J. Foresight and Innovation Policy, 9(2/3/4), Saltelli, A., Funtowicz, S., When all models are wrong: More stringent quality criteria are needed for models used at the science-policy interface, Issues in Science and Technology, Winter 2013,

32 Session in honour of Prof. Ilya M. Sobol' By Art Owen and Sergei Kucherenko END