The European Commission s science and knowledge service. Joint Research Centre

Transcription

1 The European Commission s science and knowledge service Joint Research Centre

2 Step 7: Statistical Coherence (I) Simple Correlations Pablo de Pedraza COIN th JRC Annual Training on Composite Indicators & Scoreboards 05-09/11/2018, Ispra (IT)

3 Decalogue Step 10. Presentation & dissemination Step 9. Association with other variables Step 8. Back to the indicators Step 7. Robustness & sensitivity Before Step 7: Use correlation analyses to study statistical coherence? Step 6. Weighting & aggregation Step 5. Normalization of data Step 4. Multivariate analysis Step 3. Data treatment (missing, outliers) Step 2. Selection of indicators Step 1. Developing the framework 3 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

4 Ex. Measure the performance of country s skills systems: European Skills Index Pillar Sub-pillar Indicator P1 Skills Development SP1 Compulsary training SP2 Training and tertiary education Pre-primary pupil-to-teacher ratio (students per teacher) Ind 1 Share of pop (aged 15-64) upper secondary education (%) Ind 2 Reading, maths & science scores (aged 15) (PISA score) Ind 3 Recent training (%) Ind 4 VET students (%) Ind 5 High computer skills (%) Ind 6 P2 Skills Activation SP3 Transition to work SP4 Activity rates Early leavers from training (%) Ind 7 Recent graduates in employment (%) Ind 8 Activity rate (aged 25-54) (%) Ind 9 Activity rate (aged 20-24) (%) Ind 10 Step 7. Robustness & sensitivity Step 6. Weighting & aggregation Step 5. Normalization of data P3 Skills Matching SP5 Unemployment SP6 Skills mismatch Long-term unemployment (%) Ind 11 Underemployed part-time workers (%) Ind 12 Higher education mismatch (%) Ind 13 ISCED 5-8 proportion of low wage earners (%) Ind 14 Qualification mismatch (%) Ind 15 Step 4. Multivariate analysis Step 3. Data treatment (missing, outliers) Step 2. Selection of indicators Step 1. Developing the framework 4 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

5 Ex. Measure the performance of country s skills systems: European Skills Index TOOL ρ(x,y)=cov (x,y)/σx σy To which extend two variables are linearly related 1 perfect positive linear correlation -1 perfect negative linear correlation 0 no linear correlation between the variables. Correlation coefficients within the sub-pillar, indicators with sub-pillar and with the index: r<0,70) - Avoid negative correlations - Avoid random correlations (r<0,33, The square the fraction of the variance in Y that is explained by X in a simple linear regression. 5 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations - Reflect on very high correlations (r>0.92)

6 Ex. Measure the performance of country s skills systems: European Skills Index Pearson correlations coefficients. Not significant correlations (at α = 0,01) are left blank. Very strong correlations (> 0,92) are marked in italic. 6 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

7 Ex. Measure the performance of country s skills systems: European Skills Index Pearson correlations coefficients. Not significant correlations (at α = 0,01) are left blank. Very strong correlations (> 0,92) are marked in italic. 7 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

8 Ex. 2 Global Innovation Index Global Innovation Index (GII 2017) 81 indicators 21 sub-pillars 7 pillars 2 sub-indexes (input / output) GII - Positive - Sub-pillars in their pillar - >0,70 - Explain similar amount 8 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

9 Ex. 3 Global Talent Competitiveness Index (GTCI) Global Talent Competitiveness Index (GTCI) 68 indicators 14 sub-pillars 6 pillars 2 sub-indexes (input / output) GTCI - Positive - Sub-pillars in their pillar & balance - >0,70 9 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

10 Ex 4.- Category Entrepreneurial Environment P1: Opportunity Perception Pillar P1: Opportunity Perception 1.00 P2: Start up Skills P3: Willingness and Risk P2: Start up Skills P3: Willingness and Risk P4: Networking P4: Networking P5: Cultural Support P5: Cultural Support P6: Opportunity Start up P7: Technology Sector P8: Quality of Human Resources P9: Competition P10: Voice & Agency P11: Product Innovation P12: Process Innovation P13: High Growth P14: Internationalization P15: External Financing Indicators within a dimension: Avoid negative correlations Avoid random correlations (r<0,33) Reflect on very high correlations (r>0.92) Entrepreneurial Eco-System Entrepreneurial Aspirations P6: Opportunity Start up P7: Technology Sector P8: Quality of Human Resources P9: Competition P10: Voice & Agency P11: Product Innovation P12: Process Innovation P13: High Growth P14: Internationalization P15: External Financing Pearson correlation coefficients within the dimensions of an index (normalised data after directional adjustment); Correlations less than 0.33 not significant here (blanks) 10 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

11 INDEX Entrepreneurial Aspirations Entrepreneurial Eco-System Entrepreneurial Environment Ex 4.- Table 5. Correlations between the 15 pillars, 3 sub-indices and the overall index -Eliminate Entrepreneurial Aspirations Entrepreneurial Eco-System Entrepreneurial Environment Category P1: Opportunity Perception Pillar P1: Opportunity Perception 1.00 P2: Start up Skills P3: Willingness and Risk P4: Networking P5: Cultural Support P6: Opportunity Start up P2: Start up Skills P7: Technology Sector P8: Quality of Human Resources P3: Willingness and Risk P9: Competition P10: Voice & Agency P4: Networking P11: Product Innovation P12: Process Innovation P5: Cultural Support P13: High Growth P14: Internationalization P6: Opportunity Start up P15: External Financing P7: Technology Sector P8: Quality of Human Resources P9: Competition P10: Voice & Agency P11: Product Innovation P12: Process Innovation -Weight P13: High Growth P14: Internationalization P15: External Financing -Alternative measure P1: Opportunity Perception P2: Start up Skills P3: Willingness and Risk P4: Netw orking P5: Cultural Support P6: Opportunity Start up P7: Technology Sector P8: Quality of Human Resources P9: Competition P10: Voice & Agency P11: Product Innovation P12: Process Innovation P13: High Grow th P14: Internationalization P15: External Financing Entrepreneurial Environment Entrepreneurial Eco-System Entrepreneurial Aspirations Notes: Problematic correlations are highlighted: low or close to random correlations (blue), negative correlation 11 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations Coefficients less than 0.33 (in absolute value) are not statistically significant at the 99% level.

12 Statistical coherence: Correlations 1.- How to use correlations, examples. 2.- Whats wrong with negative correlations? 3.- Whats wrong with random correlations? 4.- Other issues we may detect: - Are some indicators lost in aggregation? - Are indicators allocated in the right dimension? - Up to what level should we agregate? - Should some indicators be «over/under-weighted»? 12 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

13 2.- What s wrong with negative correlations? Sustainable Society Index 2012 Normative: conflicting issues Methodological: no clue what could happen to your average score 13 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

14 2.- What s wrong with negative correlations? X1 & X2 negatively correlated Normalization (Distance to the best performer): I1=(X1/maxX1)*100 & I2=(X2/maxX2)*100 Y. 5 I. I Country X1 X2 I1 I2 Y Rank H A B C D E F G JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

15 2.- What s wrong with negative correlations? X1 & X2 negatively correlated Normalization (Distance to the best performer): I1=(X1/maxX1)*100 & I2=(X2/maxX2)*100 Y. 5 I. I Country X1 X2 I1 I2 Y Rank H A B C D E F G JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

16 2.- What s wrong with negative correlations? The best country improves and we recalculate everything Country X1 X2 I1 I2 Y Rank H A B C D E F G Rank Only the best performer (H) improves BUT the ranking gets completely reversed and country A is last as opposed to 2 nd! 16 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

17 2.- What s wrong with negative correlations? That could happen in Sustainable Society Index PRESENT THEM SEPARATELY 17 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

18 INDEX Stability Utilization Access Availability 3.- What s wrong with random correlations? Example 1.- Index of competitiveness including a dimension related to environment. Poorly correlated no effect on the INDEX. Example 2.- Example below: One dimension poorly related to the other dimensions will be poorly related to the overall index Correlation with the index should be similar if we want each dimension to be equally important we need to increase the weight How do we communicate? Availability Access Utilization Stability JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

19 4.- Other issue we may detect Are some indicators lost in aggregation? Environmental Performance Index (Yale U., Columbia U.) Indicators, Areas & Policy objectives 19 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

20 4.- Other issue we may detect Are some indicators lost in aggregation? YES Correlation with Issue area Correlation with EPI Correlation with Policy Objective - Present them separately 20 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

21 4.- Other issue we may detect Cross correlations: Are indicators in the right dimension? Sustainable Society Index (SSI) Ideally, an indicator correlates more with its own dimension than to any other 21 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

22 4.- Other issue we may detect Up to which level should we aggregate? [Multidimensional Poverty Assessment Tool (MPAT), UN IFAD, JRC audit] 22 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

23 4.- Other issue we may detect Up to a level that is meaningful [ ] a final composite indicator should not be seen as a goal per se. It is sometimes preferred to stop the aggregation procedure at the components level and not aggregate further. This was both conceptually and statistically confirmed in the case of the MPAT. [JRC audit] 23 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

24 4.- Other issue we may detect Up to a level that is meaningful The oposite example World Justice Project: Rule of Law Index JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

25 4.- Other issue we may detect Up to a level that is meaningful World Justice Project: Rule of Law Index 2012 Finally, the eight dimensions share a single latent factor that captures 81% of the total variance. This latter result could be used as a statistical justification for aggregating further the eight dimensions into a single index by using a weighted arithmetic average. This is not currently done, as the WJP team aims to shed more light to the dimensions of the rule of law as opposed to an overall index. [JRC audit] 25 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

26 4.- Other issue we may detect Up to a level that is meaningful WJP Rule of Law Index 2014 An overall index has been calculated! 26 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

27 4.- Other issue we may detect Should some indicators be «over/under-weighted»? Summary Innovation Index (2017) 27 indicators 10 dimensions 4 groups SII SSI strongly influenced by 5 indicators 6 indicators not reflected in the SSI. SSI is an unweighted arithmetic average NOT a balance summery of its components Information loss, unless: -adjustment -use also dimension level 27 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

28 4.- Other issue we may detect Should some indicators be «over/under-weighted»? Innovation Output Indicator (IOI) Aim an output oriented measure of innovation -Patents (PCT) -Employment (KIABI) -Exports of knowledge intensive goods (COMP) -Employment dynamism (DYN) Contributions of indicators to the overall score (before and after) IOI Before re-scaling - Correlation between Composite (IOI) weights components r r^2 r^ PCT KIABI COMP_EUR DYN IOI After re-scaling - Correlation between Composite (IOI) weights components r r^2 r^ PCT KIABI COMP_EUR DYN JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

29 4.- Other issue we may detect How to communicate? Weights for the two objectives in EPI 2012: But in EPI 2010: JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations The change from EPI 2010 to EPI 2012 aimed at balancing the index

30 4.- Other issue we may detect How to communicate? p JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

31 4.- Other issue we may detect How to communicate? Scaling coefficients are not Importance coefficients AIM: Indicators explain similar amount of variance of their pillars and sub-pillar 31 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

32 CONCLUSIONS Correlations Multi-dimensionality - Redundancy Correlation Very high correlations Uni-dimensional index that is fully robust and maybe redundant Correlations is not giving specific solution but flagging problems: - Consistency of dimensions - Indicators in the right dimension - Rights Weights - Aggregation levels - Alternative measures Very low correlations Multi-dimensional index that is sensitive to the assumptions DECIDE USING ADDITIONAL INFO Robustness 32 JRC-COIN Step 7: Statistical Coherence (I) Simple Correlations

33 THANK YOU Any questions? You may contact us & Welcome to us at: The European Commission s Competence Centre on Composite Indicators and Scoreboards COIN in the EU Science Hub COIN tools are available at: