The Quality Assessment Checklist Eurostat piloting experiences

Transcription

1 The Quality Assessment Checklist Eurostat piloting experiences 10 th Meeting of the CCSA September, 2007 Madrid, Spain Håkan Linden Statistical Governance, Quality and Evaluation European Commission, Eurostat 9/18/2007 1

2 Context The Checklist is proposed to be the tool for the self-assessments to be carried out within the framework of the Eurostat evaluation activities 1. define improvement actions for key output of Eurostat to make work more efficient. Requirement stemming from Eurostat's objectives in the AMP 2. get input for the peer review of Eurostat and for Member States' NSI s ( for the principles concerning statistical areas, 7-14). Requirement in the framework of the Code of Practice 5. review priorities for statistical production. Requirement coming from the Member States NSI s The quality assessment tool to be used by the production domains that has not yet begun to report on the quality of their statistics - as take-off for their quality assessment activities - as input for the Eurostat Quality Barometer 9/18/2007 2

3 Objectives To improve the measurement, monitoring and management of data quality. To coordinate ongoing quality initiatives (process descriptions, quality reports, evaluation activities etc.). To identify strengths and weaknesses. To provide recommendations regarding improvements. To monitor implementation of improvement actions. To promote a culture of systematic quality improvement work in the ESS. 9/18/2007 3

4 The Concept of Self-Assessments Self-assessment is a comprehensive, systematic and regular review of an organisation s activities and results referenced against a model/ framework. A tool for data quality management (process quality, product quality, and user perception on product quality). Product quality is achieved through the production processes Different process designs will give priority to different product quality components The way product (and process) is perceived by user will (often) deviate from producer. Focus is on individual statistical production processes and products. Identification of improvement opportunities in processes and products. A first step towards more in-depth assessments (like audits by external organisation) 9/18/2007 4

5 Quality Assessment Checklist international organisations Task team: ECB, IMF, OECD, UNECE, ILO, FAO and Eurostat (coordinator). Builds on the original DESAP. The focus is on the processes of transformations applied by international organisations. It can be used to promote and assess the compliance with the relevant CCSA Principles Governing International Statistical Activities. A modular approach has been chosen for the design of the checklist which makes it possible to tailor it to specific needs. The assessment questions can be summarised in the assessment diagram for graphical feedback on the strengths and weaknesses. The Summary Assessment Report should be used for identifying the principal strengths and weaknesses and their importance regarding improvement. This report could be seen as a tool for managers for systematic follow-up of the implementation of the recommendations. 9/18/2007 5

6 Quality Assessment Checklist process orientation 9/18/2007 6

7 Quality Assessment Checklist assessment diagram 9/18/2007 7

8 Quality Assessment Checklist management report Summary Assessment Report Organization:. Theme/domain:. Principal strengths Principal weaknesses Recommendations regarding improvement Action Ownership Timeline [1] Status [2] [1] Implementation is foreseen in: short-term within 6 months; medium-term within over 6 months and 2 years; long-term within over 2 years [2] Already implemented; planned for implementation; not planned for implementation 9/18/2007 8

9 Piloting experiences The Checklist was piloted in two statistical domains (EU SILC and STS retail trade turnover index) in May/June The structure of the Checklist is considered flexible enough to take into account the diversity of the statistical domains of Eurostat. Some additional questions needed (CoP monitoring, implementation of CVD building blocks, Eurostat Impartiality Protocol, horizontal requirements for documentation and metadata). The workflow and the set-up of co-operation was positive (pre-filling, involvement of DDG02) Assessment of resource burden in production units (19,5h resp. 16h) The content of the Checklist is in general appropriate and relevant for the assessment of processes and outputs at Eurostat. Sufficient information were obtained for compiling the Assessment Reports and the Diagrams. The Checklist proved to be an effective tool for identifying good practices at Eurostat. 9/18/2007 9

10 Conclusions Implementation of process oriented quality assessment has to be tailored to the specific needs (size of activities, tools in place, etc.) Basic quality information are required Implementation in a stepwise approach Top- management support The set up of quality review/ audit team Establishment of feed-back mechanism 9/18/

11 Outlook DESAP- E grant in 2007 (printing of results, export results to Word or other software, compilation of short quality report, merging of results from different files etc.). Self assessment (and the Checklist) the main tool for the Eurostat quality reviews to be undertaken Revision of the ESS guidelines for quality reporting (2007/2008). Implementation strategy for the Quality Assessment Checklist for international organisations. Sharing of good/ best practices identified in the self-assessments within and between national and international organisations. 9/18/