Structural indicators and education

Transcription

1 EUROPEAN COMMISSION EUROSTAT Directorate D: Single Market, Employment and Social Statistics Unit D-5: Education and Culture Doc. ESTAT/D5/2004-ETS EN Version Structural indicators and education Meeting of the Education and Training Statistics Working Group Luxembourg, 22 and 23 January 2004

2 Structural indicators and education The objective of the present document is to inform the Education and Training Statistics Working Group on the context and latest developments of the structural indicators exercise, as well as on the related work on education and training indicators for the coming months. 1. The Lisbon strategy: the list of structural indicators At the Lisbon Special European Council held in March 2000, the Union set itself the strategic goal for the next decade: to become the most competitive and dynamic knowledgebased economy in the world capable of sustainable economic growth with more and better jobs and greater social cohesion (paragraph 5 of the Council Conclusions). The Lisbon European Council acknowledged the need to regularly discuss and assess progress made in achieving this goal on the basis of commonly agreed structural indicators. To this end, it invited (paragraph 36): " the Commission to draw up an annual synthesis report on progress on the basis of structural indicators to be agreed relating to employment, innovation, economic reform and social cohesion." In response to this request the Commission and Council agreed a list of 35 structural indicators which were approved at the Nice European Council in December The structural indicators were used in the Commission Synthesis Report, now renamed Spring Report, to the Stockholm European Council in March The indicators proved useful for illustrating areas where more policy action was needed and for measuring the progress made in reaching the Lisbon goals. The Stockholm European Council followed by the Gothenburg European Council in June 2001 identified further areas to be included in the Commissions annual Spring Report, among them sustainable development. Every year the list of indicators is reassessed taking into account political priorities as well as progress with regard to development of indicators. The new list of Structural Indicators is presented every autumn by the Commission in its Communication on Structural Indicators. Once adopted, this list is used for the Spring Report presented at the European Council meeting in the following Spring. According to the 2003 Spring European Council conclusions there was a need to enhance the quality, in particular the comparability over time, countries and regions, of statistical and analytical tools, so as to provide better analytical foundations for the design and monitoring of policies,". Responding to this request the Commission expressed its intention" in close cooperation with the European Statistical System, to report in time for the 2004 Spring European Council on how the use of structural indicators and other analytical tools for assessing progress on Lisbon strategy could be strengthened." The 2004 Spring Report will be released by the Commission in January More information concerning the Lisbon Strategy is available on the Commission Lisbon website 2

3 2. The role of Eurostat and the European Statistical System Eurostat is involved in the process of developing, selecting and improving structural indicators. In co-operation with the European Statistical System, Commission services and other data providers, among them international organisations, Eurostat provides the corresponding data and is responsible for the statistical annex to the annual Spring Report. A large part of the structural indicators are therefore based on data produced by the European Statistical System. As requested by the Gothenburg European Council the Acceding and Candidate Countries have been covered in the Spring Report 2003 for the first time. For 2/3 of the structural indicators data on Acceding and Candidate Countries including where possible an aggregate for the 10 Acceding Countries have been presented in the statistical annex. Eurostat has been working with the National Statistical Institutes of both EEA 1 Member States as well as Acceding and Candidate Countries to further improve the availability and quality of the structural indicators as well as to develop new indicators. Whereas considerable progress has been made in this work during the last years, future challenges relate to further improving timeliness, coverage and comparability. For the Spring Report 2004 it was decided to limit the number of indicators to be presented in the synthesis report. Therefore a short list of 14 indicators which make communication with the broader public easier has been selected. However the full list of Structural indicators is used for the different analyses in the framework of the Lisbon process. Data including time series and comprehensive metadata on Structural Indicators are being disseminated on the Eurostat website ( and, in more detail in the Eurostat database NewCronos. 3. Structural Indicators and Education The following indicators have been included in the present full list of structural indicators (annex 2): Under Employment em051 - Life-long learning by sex Source: Labour Force Survey Life-long learning refers to persons aged 25 to 64 who have stated that they their receiving education or training in the four weeks preceding the survey (numerator). The denominator consists of the total population of the same age group, excluding those who did not answer to the question 'participation to education and training'. Both the numerator and the denominator come from the EU Labour Force Survey. The information collected relates to all education or training whether or not relevant to the respondent's current or possible future job. 1 European Economic Area 3

4 Under Innovation and Research ir010 - Spending on Human Ressources Source: UOE data collection Public expenditure on education as a pecentage of GDP ir041 - Science and technology graduates by sex Source: UOE data collection This indicator is defined as " Tertiary graduates in science and technology per 1000 of population aged 20-29" includes new tertiary graduates in an calendar year from both public and private institutions completing graduate and post graduate studies compared to an age group that corresponds to the typical graduation age in most countries. It does not correspond to the number of graduates in these fields who are available in the labour market in this specific year. The levels and fields of education and training used follow the 1997 version of the International Standard Classification of Education (ISCED97) and the Eurostat Manual of fields of education and training (1999). ir091 - Youth education attainment level by sex Source: Labour Force Survey The indicator Youth education attainment level is defined as the percentage of young people aged years having attained at least upper secondary education attainment level, i.e. with an education level ISCED 3-4 minimum (numerator). The denominator consists in the total population of the same age group, excluding no answers to the questions 'highest level of education or training attained. Both the numerators and the denominators come from the EU Labour force survey (LFS). Under Social cohesion sc051 - Early school-leavers by sex Source: Labour Force Survey Indicator Early school leavers refers to persons aged 18 to 24 in the following two conditions: the highest level of education or training attained is ISCED 0, 1 or 2 and respondents declared not having received any education or training in the four weeks preceding the survey (numerator). The denominator consists in of the total population of the same age group, excluding no answers to the questions "highest level of education or training attained" and "participation to education and training". Both the numerators and the denominators come from the EU Labour Force Survey Indicator ir091 (Youth education attainment level) has been introduced for the first time this year in the list of structural indicators. However it is the only structural indicator related to education and training that has been included in the short list (annex 1) for the 2004 Spring Report. 4

5 The main focus for the next months will be the improvement of the quality of indicators. Activities presented and discussed during the 2004 Education and Training Statistics Working Group on education expenditure or the implementation of ISCED3 in the Labour Force Survey are part of this overall approach for the short to medium term. The European Statistical System has decided to produce a quality profile for all the structural indicators. The document presenting the rationale and the contents of this quality profile has been adopted by the Statistical Programme Committee on , and will consitute a major task for Eurostat for the coming year. The implementation of the quality profiles will start with the indicators included in the short list and will be extended gradually to cover all structural idnciators. 4. Conclusion The Structural Indicators exercise has had a significant impact on the visibility and use of the data produced by the European Statistical System. This also had as a consequence the increased demand of users for better quality data. Eurostat has worked on improving the quality of the data in the past years in very close cooperation with the ESS partners. This improvement process is progressing and it is not limited to the Short List but covers ALL the indicators in the Structural indicator list. Therefore Education and Training Statistics data providers are requested to support this effort by: - respecting the deadlines for the delivery of data, - responding to specific requests related to the Structural Indicators (e.g. related to the quality of indicators) and - providing appropriate feedback in useful time. The ETS WG members are invited to take the necessary steps to inform and mobilise accordingly the relevant bodies at the national level. 5

6 Annex 1 Structural Indicators short list for the Spring Report 2004 GENERAL ECONOMIC BACKGROUND Indicator Sub-indicators GDP per capita a1 GDP per capita in PPS Labour productivity b1 Labour productivity per person employed I EMPLOYMENT Indicator Employment rate Employment rate of older workers Sub-indicators I.1.1 Total employment rate I.1.2 Employment rate females I.1.3 Employment rate males I.1.4 Total employment rate of older workers I.1.5 Employment rate of older workers females I.1.6 Employment rate of older workers males II INNOVATION AND RESEARCH Indicator Sub-indicators II.2.1 GERD (Gross domestic expenditure on R&D) (IT expenditure R&D expenditure will be highlighted in the graph). II.9.1 Youth educational attainment level total Youth educational attainment II.9.2 Youth educational attainment level females level II.9.3 Youth educational attainment level - males III ECONOMIC REFORM Indicator Comparative price levels Business investment IV SOCIAL COHESION Indicator At-risk-of-poverty rate Regional cohesion Long-term unemployment Sub-indicators III.1.1 Comparative price levels III.7 Business investment Sub-indicators IV.2.2 At-risk-of-poverty rate after social transfers total IV.2.4 At-risk-of-poverty rate after social transfers females IV.2.6 At-risk-of-poverty rate after social transfers males IV.4.1 Dispersion of regional employment rates total IV.4.1 Dispersion of regional employment rates females IV.4.1 Dispersion of regional employment rates males IV.6.1 Long-term unemployment rate total IV.6.2 Long-term unemployment rate females IV.6.3 Long-term unemployment rate males ENVIRONMENT Indicator Sub-indicators Greenhouse gases V.1 Greenhouse gases emissions (including Kyoto Targets ) V.2 Energy intensity of the economy V.2.1 Contribution of renewables to gross inland energy consumption Energy intensity of the economy (for the time being this data will be only presented in the graph in the statistical annex) Transport V.3.1 Transport Volume of freight transport relative to GDP 6

7 Annex 2 FULL LIST of Structural Indicators Available on the internet at the address: The full Structural Indicators web site is available at the address: Indicators related to education are indicated in bold and italic General economic background eb011 - GDP per capita in PPS eb012 - Real GDP growth rate eb021 - Labour productivity per person employed eb022 - Labour productivity per hour worked eb031 - Total employment growth eb032 - Employment growth: females eb033 - Employment growth: males eb040 - Inflation rate eb050 - Unit labour cost growth eb060 - Public balance eb070 - General government debt Employment em011 - Total employment rate em012 - Employment rate: females em013 - Employment rate: males em014 - Total employment rate of older workers em015 - Employment rate of older workers: females em016 - Employment rate of older workers: males em021 - Average exit age from the labour force: total em022 - Average exit age from the labour force: females em023 - Average exit age from the labour force: males em030 - Gender pay gap in unadjusted form em051 - Life-long learning: total em052 - Life-long learning: females em053 - Life-long learning: males em Serious accidents at work: total em Serious accidents at work: females em Serious accidents at work: males em062 - Fatal accidents at work: total em071 - Total unemployment rate em072 - Unemployment rate: females em073 - Unemployment rate: males em041 - Tax rate on low wage earners: Tax rate on labour cost em042 - Tax rate on low wage earners: Unemployment trap 7

8 Innovation and research ir010 - Spending on Human Ressources ir021 - Total R&D expenditure ir022 - R&D expenditure by source of founds: industry ir023 - R&D expenditure by source of founds: government ir024 - R&D expenditure by source of founds: abroad ir031 - Level of Internet access: households ir032 - Level of internet access: enterprises ir041 - Science and technology graduates: total ir042 - Science and technology graduates: females ir043 - Science and technology graduates: males ir051 - Patents EPO ir052 - Patents USPTO ir061 - Venture capital investments: early stage ir062 - Venture capital investments: expansion & replacement ir071 - ICT expenditure: IT expenditure ir072 - ICT expenditure: Telecommunications expenditure ir080 - E-commerce: Percentage of enterprises' total turnover from e-commerce ir091 - Youth education attainment level - total ir092 - Youth education attainment level - females ir093 - Youth education attainment level - males Economic reform er011 - Comparative price levels er012 - Price convergence between EU Member States er02a1 - Price of telecommunications: local calls er02a2 - Price of telecommunications: national calls er02a3 - Price of telecommunications: calls to USA er02b1 - Electricity prices: industrial users er02b2 - Electricity prices: households er02c1 - Gas prices: industrial users er02c2 - Gas prices: households er031 - Market share of the largest generator in the electricity market er Market share of the incumbent in fixed telecommunications: local calls er Market share of the incumbent in fixed telecommunications: long distance calls er Market share of the incumbent in fixed telecommunications: international calls er033 - Market share of the leading operator in mobile telecommunication er040 - Public procurement er050 - Sectoral and ad hoc State aid er061 - Convergence in bank lending rates: Loans to households for house purchases er062 - Convergence in bank lending rates: Loans to non-financial corporations up to 1 year er063 - Convergence in bank lending rates: Loans to non-financial corporations over 1 year term loans to enterprises er064 - Trade integration of goods er065 - Trade integration of services er066 - Foreign Direct Investment intensity er070 - Business investment er081 - Business demography: Birth rate of enterprises er082 - Business demography: Survival rate of enterprises er083 - Business demography: death rate of enterprises 8

9 Social cohesion sc010 - Inequality of income distribution (income quintile share ratio) sc021 - At-risk-of-poverty rate before social transfers: total sc022 - At-risk-of-poverty rate after social transfers: total sc023 - At-risk-of-poverty rate before social transfers: females sc024 - At-risk-of-poverty rate after social transfers: females sc025 - At-risk-of-poverty rate before social transfers: males sc026 - At-risk-of-poverty rate after social transfers: males sc031 - At-persistent-risk-of-poverty rate: total sc032 - At-persistent-risk-of-poverty rate: females sc033 - At-persistent-risk-of-poverty rate: males sc041 - Dispersion of regional employment rates: total sc042 - Dispersion of regional employment rates: females sc043 - Dispersion of regional employment rates: males sc051 - Early school-leavers: total sc052 - Early school-leavers: females sc053 - Early school-leavers: males sc061 - Total long-term unemployment rate sc062 - Long term unemployment rate: females sc063 - Long term unemployment rate: males sc071 - Children aged 1-17 living in jobless households sc072 - People aged living in jobless households sc073 - Women aged living in jobless households sc074 - Men aged living in jobless households Environment en010 - Greenhouse gases emissions en020 - Energy intensity of the economy en031 - Volume of freight transport relative to GDP en032 - Volume of passenger transport relative to GDP en033 - Modal split of freight transport en034 - Modal split of passenger transport: Percentage share of cars en041 - Population exposure to air pollution by ozone en042 - Population exposure to air pollution by particulate matter en051 - Municipal waste collected en052 - Municipal waste landfilled en053 - Municipal waste incinerated en060 - Share of renewable energy (including indicative targets) en061 - Share of renewable energy (including indicative targets) en071 - Fish stocks in European marine waters en Protected Areas for biodiversity: Habitats Directive en Protected Areas for biodiversity: Birds Directive 9

10 Annex 3 Quality profile for the Structural Indicators (adopted by the Statistical Programme Committee at its 51 st meeting held in Luxembourg, ) Executive Summary Eurostat in co-operation with the European Statistical System proposes to attach a quality profile to structural indicators based on a set of useroriented quality criteria derived from the joint Eurostat and European Statistical System definition of quality as well as an assessment whether an indicator qualifies for integrated policy analysis. As far as possible and relevant a development perspective for single indicators will be outlined including an estimation of the related costs and burden for the respondents. It would be issued by Eurostat in co-operation with the European Statistical System making use of existing working structures. In a first step it would cover new indicators with the objectives to support the political selection process and gradually, the quality profile would be extended to the existing structural indicators so as to provide guidance for their interpretation. Section I provides some information on the political background and the current work related to assessing the quality of structural indicators, while section II gives an overview on the issues considered leading to this proposal. Section III presents the actual proposal of the quality profile and outlines a procedure for interaction between the political decision makers at Commission and Council level using once more already established working fora. Brussels European Council I. Background At the Brussels European Council meeting on 21 and 22 March 2003, the European Council noted the Commission s intention, in close co-operation with the European Statistical System, to report in time for the 2004 Spring European Council on how the use of structural indicators and other analytical tools for assessing progress on Lisbon strategy could be strengthened. (para 18). This proposal is intended to provide an element for this report. It proposes for any new structural indicator to undergo a formalised quality check carried out by Eurostat in co-operation with the European Statistical System. As a result, new indicators will receive a quality profile intended to provide input to the political decision making process on the selection of structural indicators on Commission and Council level. Initially, only new structural indicators will receive a quality profile. Step by step the procedure could then be extended to cover as well the existing structural indicators and as far as possible - other indicators used in related Commission exercises. 10

11 The proposal should be seen in the light of the proposal of the Commission for streamlining the working method on structural indicators adopted on October 8, 2003 as the Commission Communication on Structural Indicators 2.. In that document the Commission has proposed to cut short the list of structural indicators to be presented in the annex to the Spring Report in order to better focus public attention and promote the analysis of progress over time vis á vis the Lisbon objectives. The short list would be agreed with the Council for three years following the cycle of the main implementation exercises for the Lisbon strategy (The Broad Economic Policy Guidelines, the Employment Guidelines and the Internal Market Strategy). The current list of structural indicators will continue to be released on the Eurostat structural indicators website and in the database New Cronos. It will remain together with the short list the statistical tool to be used by the Commission when drafting the Spring Report. From Lisbon In the Lisbon European Council in March 2000 the European Union set a strategic goal for the next decade "of becoming the most competitive and dynamic knowledge-based economy in the world capable of sustainable economic growth with more and better jobs and greater social cohesion". The Council also invited the Commission to draw up an annual synthesis report (now renamed the Spring Report) on the basis of the Structural Indicators, which provide an instrument for an objective assessment of the progress made towards the Lisbon objectives, and support the key messages of the report. Following the Lisbon European Council conclusions, a set of Structural Indicators has been proposed by the Commission and subsequently agreed with the Council. This set has been subject to revisions - to reflect new policy priorities spelt out in subsequent meetings of the European Council (see below) - to replace existing indicators with better ones. The Stockholm European Council in March 2001 agreed to develop ways and means of actively involving the candidate countries in the goals and procedures of the Lisbon strategy. It extended the Lisbon strategy by deepening its social dimension and requested the development of indicators to monitor quality at work and social inclusion (paras 9, 27-29). In Gothenburg the European Council on June 2001 agreed a strategy for sustainable development which completes the Union's political commitment to economic and social renewal, added a third, environmental dimension to the Lisbon strategy and established a new approach to policy making (para 20). Moreover, it announced that starting from Spring 2003, the Commission will begin covering the candidate countries and their national policies in its annual Spring Report (para 11). to Barcelona As regards the development of the indicators, co-ordination between different processes related to the establishment of indicators is explicitly 2 COM (2003) 585 final 11

12 Task of the structural indicators Principles guiding the selection of structural indicators Current annual producer-oriented quality assessment of structural indicators to be supplemented by an user-oriented quality reporting. being called for by the Barcelona European Council on March 2002 (paragraph 27). As the Commission s Spring Report has included the candidate countries as from 2003 onwards, it is mentioned as important for the candidate countries to take the policies and objectives of the Community s strategic sustainable development policy into account in the period prior to accession, so that their situation is reflected in the Spring Report for 2003 (paragraph 48). Following the European Council conclusions the key objective of structural indicators is as follows: to support assessment of the EU-Member States and Candidate Countries progress towards the Lisbon objectives compared to important competitors (US and Japan at least) within a ten years time frame starting from 2000 and to underpin the analysis presented in the Spring Report. It is against these objectives that the selection of single indicators as well as of the set of structural indicators has to be evaluated. In its autumn 2001 Communication on structural indicators 3 the Commission defined 7 criteria for the choice of the indicators: They should be (1) easy to read and understand; (2) policy relevant; (3) mutually consistent; (4) timely available; (5) comparable across Member States and as far as possible with other countries; (6) selected from reliable sources; and (7) should not impose too large a burden on Member States and respondents. There are additional principles that have guided the Commission in defining the set of structural indicators: The list should be stable to allow for the measurement of progress over time and for a process of continuous improvement of the indicators in terms of reliability and quality. There is also room for flexibility in the list of indicators to incorporate new priorities and improved indicators which have become available in the meantime. The list should be kept short in order to send clear, simple and focussed policy messages but it should also be balanced between the domains covered. Starting from the first set of indicators Eurostat has carried out an annual quality assessment of structural indicators based on the joint Eurostat and European Statistical System definition of quality in statistics 4. This producer-oriented assessment draws from more detailed quality reporting within the single Eurostat data domains. Using a rather technical approach, it is directed towards the producers of statistics and intended to stimulate and monitor over time quality improvements of existing indicators within the European Statistical System. Results have been regularly presented to the Statistical Programme Committee. This proposal centers on supplementing the current approach by a quality screening focusing on user-oriented quality criteria directed towards the users rather than the producers of structural indicators. It is to be used as an input to the political selection process of new indicators as well as to guide 3 4 COM (2001) 619 The following dimensions of quality are covered: framework conditions, relevance, accuracy, timeliness, clarity and accessibility, comparability, coherence and completeness. 12

13 the use and interpretation of structural indicators. It will as far as possible draw from existing quality reporting established for structural indicators as well as in the single domains. II. Issues Proliferation of indicators calls for more user guidance and an analytical foundation. Role of the European Statistical System to be reinforced in the process of indicator selection Statistical information is increasingly used for policy making at EU and national level and will continue to play an important role as the new method of open co-ordination will be extended to more policy areas and to cover Candidate Countries. This calls for a more rigorous and transparent approach towards data quality. In particular, users at European and national level will have to be better informed on the suitability of an indicator to compare countries rather than policies and on any mid-/long-term requirements of harmonisation of the underlying data sources. Structural Indicators have received a high level of public attendance ranking among the top in terms of Eurostat website hits. Although the indicators are well documented, providing an example of Eurostat metadata policy, some supplementary guidance for users of structural indicators giving a quick overview on strengths and limitations of the underlying data would improve user-friendliness of this highly diverse product. The mechanisms through which indicators are currently selected (interaction between Commission and Council) lead to partly unsatisfactory results. Whereas some indicators have been criticised on the grounds of data quality, the set of structural indicators as such has been criticised for its sheer quantity, for a lack of a systematically applied internal logic and for its shortcomings to fully meet the needs of a sustainable development strategy 5. Thus there is a need for an extension of the current quality approach to incorporate as far as possible an evaluation of the set as well as an assessment of the potential to integrate indicators in a common framework like the European Statistical System s accounting systems to strengthen the analytical basis for the structural indicators. Whereas Structural Indicators have created a momentum for quality improvements of the existing indicators and the development of new indicators within the European Statistical System and by Candidate Countries, there is a need to apply some kind of filter in order to avoid overburdening the statistical systems. The current political decision mechanism makes it difficult to balance political and technical assessment of indicators. This calls for a systematic involvement of the European Statistical System to qualify the indicators in order to limit inappropriate use. This seems to be of particular importance given the variety of sources (official and unofficial) from which the indicators stem and their related different levels of reliability. 5 See e.g. Opinion of the European Economic and Social Committee on The Lisbon Strategy and Sustainable Development (2003/C95/14) or Spangenberg, J.H. (2003) Integrating economic, social and environmental policies: who calls the tune? A background paper for the EU Summit 13

14 to support the political decision. In order to be useful, an indicator or a set of indicators has to be recognised and accepted. Thus, a more formal involvement of the national statistical institutes of the Member Countries would facilitate decision making at Council level. However, technical quality criteria should guide, not replace the political decision. III. Proposal An user-oriented quality profile to be issued by the European Statistical System Quality profile Eurostat proposes an user-oriented quality profile based on the initial objectives of structural indicators as defined by the European Council meetings. The proposal aims at - providing input to the political selection process of any new structural indicators - systematic involvement of the European Statistical System at an early stage thus releasing as far as possible the Council level from technical discussions and promoting identification of potential European Statistical System s alternatives - promoting transparency as regards necessity of further development work and if possible an estimation of the related costs. - promoting the selection of indicators which enable an integrated policy analysis in line with the implementation of the Commission new Impact Assessment 6 in the framework of the Better Regulation Action Plan 7. It is intended to extend this quality profile in the mid-term to also cover the already existing indicators, thus supplementing Eurostat metadata provided for structural indicators on the Eurostat Structural Indicators website 8 as well as the Eurostat data base New Cronos (STRIND domain). 1. Quality profile It is being proposed that during the decision-making process on a new structural indicator, the European Statistical System in line with the procedure described under 2. issues a quality profile for any new indicator (see Annex I) covering the dimensions listed below. The proposal focuses on structural indicators and quality with regard to their specific user demands only, i.e. essentially the Commission Spring Reports to the European Council. Thus, it is based on the quality criteria for the selection of new structural indicators as mentioned in the Commission Communication of autumn and tries to put them into more concrete terms. At the same time it supplements the annual quality assessment carried out by Eurostat for the existing structural indicators by a more useroriented description. The criteria used in the quality profile are derived from the joint Eurostat and European Statistical System definition of COM (2002) 276 see document COM (2002) COM(2002)

15 quality in statistics and have been tailored to the characteristics being most important to users of structural indicators given their main objectives. It should be noted that for other purposes the quality concepts applied may have to be chosen differently or may yield different results. Recalling the objectives of structural indicators mentioned above, the following user-oriented quality criteria can be identified: 1.1 Relevance Objective and relevance of an indicator The indicator should be the best available for the purpose mentioned. It should be easy to read and suited to being interpreted normatively in a clear way. This will not form part of the quality rating but each quality profile will give an explanation of the policy relevance and context of an indicator, including information on possible restrictions with regard to describing the characteristics of interest. 1.2 Feasibility criteria 1.3 Technical criteria Data availability (coverage/timeliness) The indicator has to be available in time for Member States, Candidate Countries and as far as possible the United States and Japan. It should cover in principle two target reference years, of which one is the Lisbon year 2000 and the other, if possible, the year preceding the Spring Report. Moreover, longer time series are required to allow for a dynamic analysis. Overall accuracy The indicator should stem from reliable sources meeting high standards as regards the technique and methodology applied, errors, etc. Comparability across countries The indicator has to be comparable between Member States, Candidate Countries as well as with the United States and Japan. Comparability over time The indicator has to be comparable from one year to another starting at least with Systemic criteria Contribution to quality of the set/potential to qualify for integrated policy analysis Following the Lisbon and subsequent European Council objectives, the following domains have been identified as areas to be covered by structural indicators: employment, innovation and research, economic reform, social cohesion, environment as well as some key information on the general economic background. In order to get an indication of whether any new indicator adds value to the existing set of structural indicators, any new indicator will be assessed on whether it provides new information compared to the current set of structural indicators, including, as far as possible, an assessment of whether an indicator provides information on more than one domain or even on possible spill-over effects between the priority policy areas. Moreover, indicators will be scrutinised whether they qualify for an integrated policy analysis provided through European Statistical System s accounting frameworks. This would act as a guide to decision-makers and would be 15

16 included in the quality profile, but would not form part of the quality rating. 1.5 Other restrictions to quality 1.6 Quality grades 1.7 Quality development work and costs Procedure for issuing the quality profile by Eurostat in close cooperation with the European Statistical System making use of existing working for and existing Other restrictions In addition to these criteria room has been provided to add any further reservations which can be expected to seriously limit the use of an indicator in the Spring Report. They may relate e.g. to the complexity of an indicator, a lack of an unambiguous scientific basis or to the coherence with other existing indicators, lack of comprehensive metadata etc. Overall assessment The quality profile will provide an overall quality assessment of each indicator according to 4 quality grades (AA, A, B,C). A fifth category has been added to qualify an indicator as to be (further) developed before entering the list of structural indicators. Annex II describes the quality grades in detail. Development perspective As far as necessary (and possible) a first indication of further development work will be given, in particular for the indicators graded B or C for which quality improvements are most pressing. If possible it will be supplemented by a qualitative estimation of the related costs in terms of implications for the Member Countries and burden for the respondents. 2. Procedure In order to make this proposal become operational for new indicators it is foreseen to include more systematically the European Statistical System in the political consultation procedure. The proposal is meant to be implemented by fully drawing on existing working structures that have been established within the European Statistical System as well as on Commission and Council level to deal with the production, development or selection of structural and other indicators. It should be pointed out that in most areas it already has become good practice that European Statistical System representatives advice on feasibility and quality of indicators based on European Union harmonised statistics. This practice is to be broadened to all actions that require the production of indicators. Usually this dialogue is established in the framework of Council Committees Indicators Subgroups or related to the implementation and monitoring of European Union policies. Commission services proposing any new indicator would consult at an early stage, i.e. during spring of the year preceding the Spring Report, Eurostat, for the development of that indicator. Once Commission services consider an indicator ripe to be proposed as a new structural indicator, Eurostat, in co-operation with the European Statistical System will issue a quality profile for the indicator. To this end a detailed questionnaire covering the quality dimensions will be sent to the National Statistical Institutes using existing European Statistical System working structures. Moreover, the questionnaire will as far as possible draw on existing information on quality in the single data domains. On the basis of the replies the quality profile will be filled in and a 16

17 information and promoting an early agreement on the final list of structural indicators. grade will be given. This procedure be applied for indicators for which the national data are provided by the National Statistical Institutes and/or by other official sources, thus emphasising the co-ordination role of the National Statistical Institutes within their countries. For indicators stemming from unofficial sources, Eurostat will undertake to contact the source on information about quality and collaborate with the National Statistical Institutes to fill in the quality profile. It should be noted that this assessment can only be applied on the basis of available data. Thus, unavailability of data in time may result in the indicator being considered as to be further developed. Once this indicator is agreed to be released in the Eurostat database of structural indicators, the quality profile will be attached to the indicator. The quality profile may be modified or updated on the basis of major changes with regards to the underlying data or the quality information available. In principle, the same procedure would be applied to any new indicators proposed on Council level. The Council might wish to conclude that it recommends to limit proposals for additional new indicators to the period before October to allow for the quality assessment procedure to be launched well in advance before the list of structural indicators is to be agreed. Furthermore the Commission notices good practice of some Council Committees working on indicators to engage in an active dialogue with the European Statistical System on technical feasibility and quality of indicators in the preliminary stages of decision making. The Council might wish to conclude that a systematic involvement of the European Statistical System in the development of indicators is being considered desirable in order to exploit to the largest extent possible existing official data sources. It is proposed, for the mid term, to fill in the quality profile as well for the existing indicators, thus extending the Eurostat metadata provided so far by some guidelines on how to interpret the respective indicator and by information on the strengths and shortcomings of an indicator that should guide its use. Step by step a similar quality profile could be extended to also cover other indicators used in the related Commission exercises, thus promoting overall coherence of the indicators used at Commission level as well as transparency as the users will be put in a position of having a general idea of the quality of the underlying data sources and the limitations of use of the indicator in question However, it should be kept in mind that the proposed quality profile is tailored to the specific user needs and objectives of structural indicators. 17

18 Annex I STRUCTURAL INDICATORS QUALITY PROFILE Indicator Responsible COM DG: Eurostat Unit: Date European Statistical System Working Group 1. Objective and relevance of the indicator * : If necessary, comments on restrictions of its relevance: 2. Data availability Member States Candidate Countries US and Japan EEA-EFTA Target Reference T 1 years for SR 200 T 2 Comments (including information on time series: 3. Overall accuracy Comments High Restricted (sources, errors, methodology etc.) 4. Comparability across countries Comments High Restricted 5. Comparability over time Comments High Restricted 6. Contribution to quality of the set/potential to qualify for an integrated policy analysis* Comments/ Assessment: 7. Other characteristics which may lead to Comments restriction for using this indicator in the Spring Report: _ 8. Overall assessment 9. Development perspective for improving quality of this indicator (including as far as possible an indication of burden on Member States and respondents.) AA A B C Indicator to be developed * Not part of the quality rating 18

19 Description of Quality grades Annex II Grade AA Grade A Grade B Grade C Indicator to be developed An AA is given if the indicator fulfils the requirements listed for the A grading and in addition (1) time series are available (starting from 1991 for the Member States and 1995 for the Candidate Countries) and (2) data for at least reference year t-2 (real data for a majority of countries) is available in time for the Spring Report. An indicators is graded A if data is available on time for the Spring Report of the year t for Member States, at least acceding countries, US and Japan. Data cover the years 2000 until at least the reference year t-2 (including estimates). The underlying data is collected on the basis of a common methodology for the European Union with the Candidate Countries following the same approach. Data for US and Japan can be considered comparable with any major differences being assessed and documented. Data are comparable over time; impact of procedural or conceptual changes being documented. Data is collected from reliable sources applying high standards with regard to methodology/accuracy and is well documented in line with Eurostat metadata standard. An indicator is graded B if data is available on time for the Spring Report of the year t for most Member States and at least most of the acceding countries. Data cover the years 2000 until at least the reference year t-3. There are some serious shortcomings with regard to either comparability between Member States/Candidate Countries/US and Japan (including the lack of data) or break in series for several countries which seriously hamper comparison over time. Deficiencies with regard to assessing and documenting impact of these shortcomings might be identified. Data is collected from reliable sources applying high standards with regard to methodology/accuracy and is well documented in line with Eurostat metadata standard. An indicator is graded C if data coverage on time for the Spring Report of the year t has some severe deficiencies as regards country coverage and/or timeliness. There might be either some serious shortcomings with regard to comparability between Member States/Candidate Countries/US and Japan (including the lack of data) or break in series for several countries which seriously hamper comparison over time. Data might have to be interpreted with care as methodology/accuracy does not meet high quality standards. This indicator is not ready to be used for the Spring Report. Data availability in time for enough countries and/or with a sufficiently high timeliness seems to be unlikely. There might be either some serious shortcomings with regard to comparability between Member States/Candidate Countries/US and Japan (including lack of data) and/or break in series for several countries which seriously hamper comparison over time. Data might have to be interpreted with care as methodology/accuracy does not meet highest quality standards. 19