Quality in statistics, the quality framework of the ESS Zsuzsanna Kovács Team Leader, Quality Team Unit D4, Eurostat 24/10/2016

Transcription

1 Quality in statistics, the quality framework of the ESS Zsuzsanna Kovács Team Leader, Quality Team Unit D4, 24/10/2016

2 Overview What is quality and quality in statistics? The European Statistical System (ESS) Quality framework of the ESS The ES Code of Practice and Quality Assurance Framework of the ESS, the 4 levels of quality assurance Quality reporting in the ESS Quality assurance methods and tools, continuous improvement The Quality Declaration of the ESS From quality assurance to a holistic view of quality 2

3 What is quality? A relative, multi-dimensional concept Different definitions and approaches exist The most general and succinct definition: "fitness for purpose/use" Definitions of international standards: ISO 9000: "Degree to which a set of inherent characteristics fulfils requirements" ISO 8402:1986: "Totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs" 3

4 What is quality in statistics? Encompasses all aspects of how well statistical processes and outputs fulfil expectations of all of their users and stakeholders => Good quality of statistics is: Not just meeting user needs but also Addressing respondent concerns regarding reporting burden and confidentiality Ensuring institutional environment is impartial, objective, comprising sound methodology and cost-effective procedures 4

5 The European Statistical System (ESS) a partnership The European Statistical System (ESS) is the partnership between the Union statistical authority, which is the Commission (), and the national statistical institutes (NSIs) and other national authorities (ONAs) responsible in each Member State for the development, production and dissemination of European statistics. 5

6 Quality framework of the ESS The current 'common' quality framework of the ESS is composed of cf. in the Preamble of the ES Code of Practice The European Statistics Code of Practice The general quality management principles quality for the institutions as a whole, QUAL Action 3 The Quality Assurance Framework of the ESS, as the 'extension' / complement of the ES Code of Practice The quality framework of the ESS complements the legal framework and they both constitute the robust framework in which high-quality European statistics are developed, produced & disseminated 6

7 ES Code of Practice aim Sets the standards for developing, producing and publishing European statistics Improves trust and confidence in the independence, integrity and accountability of the statistical authorities as well as in the credibility and quality of the statistics they develop, produce and disseminate Promotes the application of best international statistical practices, principles and methods by all producers of European statistics 7

8 ES Code of Practice: the ESS concept of quality in statistics Quality is defined by the ES Code of Practice in terms of the Set-up of the institutional environment (6 principles) Statistical outputs (5 principles) Statistical processes (4 principles) 8

9 Institutional environment/conditions Are the "prerequisites of quality" in IMF's Data Quality Assessment Framework Within the institutional environment a programme of statistical processes is conducted => Some of the components can be considered as being part of the process quality components as well and conversely (dual applicability) 9

10 Institutional environment/conditions Principles or components: Professional independence Mandate for data collection Adequacy of resources Commitment to quality Statistical confidentiality Impartiality and objectivity 10

11 Statistical output Principles, i.e. the different quality criteria / dimensions / components: Relevance Accuracy and reliability Timeliness and punctuality Coherence and comparability Accessibility and clarity 11

12 Quality criteria of the statistical output OECD: ESS: ECB: IMF: FAO: UNESCO: UNECE: relevance, accuracy, credibility, timeliness (and punctuality), accessibility, interpretability, coherence (within dataset, across datasets, over time, across countries) relevance, accuracy, timeliness and punctuality, accessibility and clarity, coherence (within dataset, across dataset), comparability (over time, across countries) accuracy/reliability, methodological soundness, timeliness, consistency prerequisites of quality, accuracy and reliability, assurances of integrity, methodological soundness, serviceability (timeliness and periodicity), accessibility, serviceability (within dataset, across dataset, over time, across countries) relevance (completeness), accuracy, timeliness, punctuality, accessibility, clarity (sound metadata), coherence, comparability relevance, accuracy, interpretability, coherence relevance, accuracy (credibility), timeliness, punctuality, accessibility, clarity, comparability (across datasets, over time, across countries) 12

13 Statistical processes Output quality is achieved through process quality Process quality has 2 (or 3) broad aspects: Effectiveness: leads to the outputs of good quality which meet or exceed users and stakeholders' expectations Efficiency: leads to production of outputs at minimum costs to the statistical office and to the respondents who provide the original data, without wasting resources (Economics: generates requisite revenues from the process so that the organisation can be sustained) 13

14 Statistical processes Principles or components of individual statistical processes: Sound methodology Appropriate statistical procedures Non-excessive burden on respondents Cost effectiveness 14

15 ES Code of Practice mechanism Self-regulatory Applies to, National Statistical Institutes of the ESS and other national authorities producing European statistics For each Principle, there are Indicators showing how compliance can be demonstrated 15

16 ES Code of Practice example Principle 8 Appropriate statistical procedures, implemented from data collection to data validation, underpin quality statistics Indicator 8.6 Revisions follow standard, well-established and transparent procedures 16

17 ES Code of Practice monitoring National and self-assessments in ESS peer reviews in (1 st round) Annual monitoring and reporting of improvement actions ( & ESGAB) 2 nd round of self-assessments and ESS peer reviews in Annual monitoring and reporting of improvement actions: just launched for the ESSC meeting of November

18 Quality Assurance Framework (QAF) of the ESS Example Aim 3 rd level of quality assurance Applicable across all statistical domains But it is not part of the Code 18

19 Quality Assurance Framework of the ESS example Principle 8 Appropriate statistical procedures, implemented from data collection to data validation, underpin quality statistics Indicator 8.6 Revisions follow standard, well-established and transparent procedures Methods of implementation Guidelines on revision of published statistics exist, are applied and made known to users Revisions accompanied by explanations made available to users Quality indicators on revisions are calculated and published 19

20 Quality Assurance Framework of the ESS aim Provides methods and tools at institutional and process level Contains also links to relevant reference documentation Can also be considered as guidance to compliance assessors Is a living document, to be developed further at a later stage Does not address process-specific issues

21 Process-specific quality assurance 4 th level Level 1 = CoP: Principles (standards) Level 2 = CoP: Indicators (how the standards can be demonstrated) Level 3 = Quality Assurance Framework (what methods and tools can be used) Level 4 = Process-specific quality assurance, adapted to the needs of the process (e.g. excessive deficit procedure statistics)

22 Quality reporting in the ESS Member States have to provide with reports on the quality of the data transmitted has to assess the qu ality of the data and has to prepare and publish reports on the quality of European statistics Modalities, structure and periodicity of quality reports depend on the sectoral legislation in the different statistical domains But the general framework for quality reporting in the ESS is standardised: ESS Handbook for Quality Reports 22

23 ESS Handbook for Quality Reports Describes the structure and content of a standard, detailed ESS quality report Contains practical examples as well Explaines the standard ESS quality indicators and their use Is now being revised, to include the latest developments / /KS-GQ EN-N.pdf 23

24 Recent developments: quality reporting using metadata facilities Reference metadata structure of Euro-SDMX Metadata Structure (ESMS): basic quality information => short or "user quality report" For a more detailed information on quality: the ESS Standard for Quality Reports Structure (ESQRS) "producer report" They both constitute the Single Integrated Metadata Structure (SIMS) which is the quality reporting standard of the ESS and They are used in the creation and exchange of quality and metadata reports in the Member States and, making the reports standardised and comparable 24

25 SIMS, version

26 Standard ESS quality indicators (QPI) R1: Data completeness rate* (S ) AC1: Data tables consultations (S ) A1: Sampling errors indicators (S ) AC2: Metadata consultations (S ) A4: Unit non-response rate (S ) AC3: Metadata completen. rate (S ) A5: Item non-response rate (S ) A2: Over-coverage rate (S ) TP2: Time lag final results (S ) A3: Common units proportion (S ) TP3: Punctuality delivery&publ.* (S ) TP1: Time lag 1 st results (S ) CC2: Length of comparable T series (S ) CC1: Asymmetry for mirror flows (S ) A6: Data revision average size (S ) A7: Imputation rate (S ) 26

27 Quality assurance methods & tools, continuous improvement User requirements Standards 3- Conformity Labelling 2- Evaluation Quality assessments 1- Documentation and measurement Quality reports and indicators Process descriptions User satisfaction surveys Quality improvements Statistical products Production processes User perception 27

28 The Quality Declaration of the ESS The 'crowning' of many years' work to demonstrate the competitive advantage of European statistics, what makes them different from other data 28

29 Why in the form of a 'Declaration' Because It is a self-commitment, a continuous self-engagement of all members of the European Statistical System (ESS) It builds on all the efforts that the ESS deployed in order to guarantee the development, production and dissemination of high-quality European statistics and services (amended Regulation 223, second round of peer reviews, ) 29

30 Its main message: awareness A very high level of quality awareness exists in the ESS All members of the ESS partnership do the same in terms of quality & The standards are set very high, as defined in the world-class quality framework: European Statistics Code of Practice => 30

31 and official guarantee: 'ESS made' Therefore, the product: European Statistics equals high quality and can be trusted is independent and 'unique' in today's oceans of information is based on sound methodologies aims at minimising burden on respondents is relevant and anticipates user needs is based on close cooperation with stakeholders is decided upon in a democratic manner provides high value for money for the societies 31

32 Quality for the institutions as a whole difference between managing and assuring quality Quality management is the set of systems and frameworks which are in place within an organisation to manage the quality of products and processes a broad range of activities Quality assurance is the guarantee of an organisation that the product or service it offers meets the accepted quality standards; according to the ISO, it is a part of quality management a more technical part, about the core business 32

33 in their respective frameworks Quality management is implemented via the Quality Management Systems: a set of interrelated or interacting elements that organisations use to direct and control how quality policies are implemented and quality objectives achieved Quality assurance is implemented via the Quality Assurance Frameworks (QAF): an umbrella for quality work at a statistical office, a place for recording quality concepts, policies and practices and a basis for creating and promoting quality culture 33

34 Quality Management Systems "A set of co-ordinated activities to direct and control an organisation in order to continually improve the effectiveness and efficiency of its performance" EFQM ISO 9000 series Lean Six Sigma Balanced Score Card, etc 34

35 Quality in the ESS Summary Based on: Regulatory, legal framework: Legislation (Treaty and Regulation) Self-regulatory mechanisms: Code of Practice and Quality Assurance Framework Monitoring quality (quality reporting, peer reviews ) Extending the approach from focussing only on the core business (statistical production) to a holistic view of quality 35

36 Questions / comments? Thank you for your attention 36