Jiri Krovak - CZSO, Czech Republic

Transcription

1 SESSION 3 Quality Assurance chair Jiri Krovak - CZSO, Czech Republic The Use and Convergence of Quality Assurance Frameworks for International and Supranational Organisations Compiling Statistics Antonio Baigorri, Håkan Lindén - Eurostat (European Commission), Luxembourg In recent years, much work has been going on in the field of quality management. Various quality management models and frameworks (like the EFQM model, ISO 9000, the DQAF of the IMF, or the European Statistics Code of Practice) have been advocated for use in organisations producing and disseminating statistics. In considering data quality, statistical authorities have worked extensively on more operational definitions of quality, in particular for assessing the statistical output quality, and today most organisations do agree that quality is about the ability of a product or service to satisfy stated or implied users needs. It has also become apparent that good process quality is a precondition for high output quality and there has been considerable attention during recent years towards identifying, describing and measuring the quality of the statistical processes. Moreover, it is widely acknowledged that the institutional setup in which a statistical authority operates can have a significant impact on the data quality. Much of the work on quality undertaken by international and supranational organisations has centred on the development of quality frameworks and processes for measuring and improving the quality of data compiled and disseminated by statistical authorities in (member) countries. However, it is only recently that international organisations started to apply to themselves the quality approaches they recommend to national statistical authorities. The current situation, where most international organisations do not have formalised quality frameworks in place and the fact that the existing frameworks are different (albeit overlapping), has been recognised by the Committee for the Coordination of Statistical Activities (CCSA). In order to promote the use and convergence of quality frameworks for international organisations, the CCSA supported in 2005 a Eurostat coordinated project on the use and convergence of international quality assurance frameworks with the aim of bringing different quality initiatives under a common framework in order to ensure that the right quality assurance procedures (methods and tools) are put in place and that the current and future quality activities of international organisations are well integrated. In this paper, we review the main results arising from the project, stressing elements that are important for facilitating the implementation of quality assurance frameworks in international organisations. It presents the scope and uses of quality assurance frameworks; the impact on data quality; costs and benefits; relationships between frameworks; implementation experiences; monitoring and evaluation; and how to work with quality tools in order to facilitate a systematic implementation of data quality assessment in international organisations. Finally, we discuss the importance of carefully tailoring a quality assurance activity to the needs of each international organisation taking into account aspects such as the institutional environment, the quality management system in place, the size of the statistical activities, and the resources available. antonio.baigorri@ec.europa.eu, hakan.linden@ec.europa.eu QUALITY MANAGEMENT QUALITY ASSURANCE QUALITY MEASUREMENT 19

2 QUALITY ASSESSMENT QUALITY ASSURANCE FRAMEWORK SELF-ASSESSMENT Recent Quality Related Initiatives at Statistics Canada Claude Julien - Statistics Canada Statistics Canada has a long history of sound data quality management practices. Its Quality Assurance Framework (Statistics Canada, 2002) defines quality based on the concept of fitness for use and describes the mechanisms by which it communicates with its users, plans its activities, designs its statistical programs, disseminates its information and reports on its programs. At the Q2006 conference, Julien and Born (2006) presented options for adding a quality management assessment process to the reporting mechanism. In August 2006, Statistics Canada reported that it had discovered an error in its Consumer Price Index. While the magnitude of the error was small, it was the third error in data released by the Agency within an 18 month period. This sequence of events resulted in two initiatives. First, a quality review of nine mission-critical programs was launched in September 2006 and completed in March 2007 (Statistics Canada, 2007). Julien and Royce (2007) presented the methodology and main outcomes at the Third International Conference on Establishment Surveys (ICES-III). The second initiative was the development of a quality assurance training exercise for all staff involved in managing the production of statistical programs. The review of the mission-critical programs led to the establishment of an ongoing Quality Review Program (QRP). Every year, the QRP will conduct an independent internal review of selected statistical programs on well-defined aspects of their processes. The number of programs under review will depend on the needs of the Agency at the time. The QRP will be an integral component of the Agency s Quality Assurance Framework. The programs under review will be identified within an existing priority-setting mechanism based on the corporate knowledge of those programs that are more at risk of experiencing a quality incident or that would, for other valid reasons, benefit the most from such a review. The results of the review will be used to allocate resources to strengthen programs where weaknesses are found, as well as to identify best practices that can be shared with other programs. Furthermore, by focusing on the programs processes, rather than on their design or outputs, the QRP will supplement the existing reporting mechanism. The presentation will focus on the quality reviews. It will provide an overview of the review of the mission-critical programs. It will describe the Quality Review Program, how it will be integrated with other mechanisms of the Quality Assurance Framework and how it will interact with the quality assurance training exercise. References Julien, C. and Born, A., (2006). Quality Management Assessment at Statistics Canada, Proceedings of the European Conference on Quality in Survey Statistics (Q2006), Julien C. and Royce D. (2007), Quality Review of Key Indicators at Statistics Canada, Third International Conference on Establishment Surveys, to be published Statistics Canada (2002), Quality Assurance Framework, No XIE , Statistics Canada (2007), Review of quality assurance practices, No XWE , claude.julien@statcan.ca 20

3 Reducing Burden while Increasing Quality at a Government Agency David A. Marker, Mary Dingwall - Westat, USA Marla D. Smith - U.S. Environmental Protection Agency Since 1979, the United States Environmental Protection Agency (USEPA) has used its Quality System to manage the quality of its environmental data collection, generation, and use. The primary goal of the Quality System is to ensure that the environmental data are of sufficient quantity and quality to support the data's intended use. As part of its Quality System, USEPA creates and applies quality assurance project plans (QAPP) to its environmental programs. Each QAPP describes and documents primary data collection, secondary data usage, and data processing (such as modeling). The purpose of the QAPP is to describe what will be done at each step to assure quality, including who is responsible for each activity. This presentation describes a recent in-depth assessment of quality of one of USEPA s programs. The USEPA staff and contractors evaluated current practices to streamline and improve quality planning and QAPPs so that they focus on the activities that truly affect quality. This assessment included getting input from a wide range of people; identifying the steps in the process that can affect quality; determining what needed to be considered at each of those steps; and simplifying the required format for the actual QAPP. The end result is a streamlined process that improves quality planning while reducing the burden to develop QAPPs. While QAPPs may be unique to USEPA, the steps of understanding the processes that are key to quality improvement, gaining buy-in from all levels of staff, and putting into place a user-friendly process are lessons that have wide-ranging application across all government agencies. QUALITY MANAGEMENT PROCESS QUALITY CURRENT BEST METHODS Embedding Knowledge Management in the QMS Mária Dologová - Statistical Office of the Slovak Republic Ján Dolog - EOQ Senior Consultant for Quality Management Systems, Slovakia Transition from the information to knowledge society is reflected practically in all organisations; especially in organisations offering services and sophisticated products. This also concerns the NSIs that are, considering their character, knowledge organisations. The challenging environment characterised by driving forces like the need for statistical knowledge products / services and for ensuring convenient access to them, changes in the field of statistics (procedures, methods), need to exchange knowledge within ESS, need for intensive development of statistical offices, labour force mobility and seniority - threat of loosing knowledge, fast development of ICT, etc. requires statistical offices to ensure both: the high level of knowledge needed and the ability to make the knowledge work for the organisation. From this perspective the knowledge has to be seen here as the key resource and the managing knowledge as the core competence of the statistical office. The NSIs generally are aware of these facts and there are several knowledge initiatives that confirm it. To achieve the desired results however, it is inevitable to implement a conscious and systematic management of knowledge processes. It means to integrate the knowledge management into the QMS itself as the basic precondition for its own design and improvement and thus for the superior performance and the sustainable development of the NSI itself. The implementation of this approach has also started in the Statistical Office of the Slovak Republic (SOSR). Decision on the knowledge management implementation The decision to implement the knowledge management has been reflected in the main strategic statements of the organisation in the vision, mission and in the 21

4 QUALITY MANAGEMENT SYSTEM KNOWLEDGE ORGANISATION KNOWLEDGE MANAGEMENT common shared values. The concrete decision was formulated in the strategic objective of the SOSR for the area of knowledge management: to implement the knowledge management with the aim to support the strategic objectives of the institution, with the special focus on creation of knowledge products. This should especially be realised through ensuring access to the knowledge needed/resulting in/from the processes via an effective sharing of knowledge. Approach to the knowledge management implementation In the following step the dimensions of the knowledge management in the SOSR were assessed and the SWOT analysis elaborated. On the basis of the results the main objective for the knowledge management implementation was formulated. In accordance with the SOSR QMS it was naturally decided to focus on the statistical processes as the value-adding process of the institution, i.e. on the process that directly contributes to creation of value for customer - here via creation of the knowledge product. Strategic objectives formulated for other areas create the general framework for the realisation of the main objective. The framework especially covers the need for transition from data to information and knowledge products, from dissemination to convenient access, as well as the need for developing partnerships. The realisation of the main objective covers: identification as well as the acquisition, organisation, sharing, use, reuse of the knowledge within the value-adding process (based on the knowledge audit and gap analysis), improvement of the knowledge sharing culture through the human resources development (impact on all parts of the HR management cycle and on the common shared values), support of the knowledge sharing via suitable organisational forms/structures (in the intra- and inter-organisational context) and premises, support of the knowledge process via suitable ICT (for the organisation of explicit knowledge, facilitating communication, discovering hidden knowledge visualisation of outputs statistical products, etc.). Improvement of visibility and integration of knowledge management into the QMS standards In the final part the impact of the knowledge management implementation on the processes of the institution is reflected in the QMS, it means the impact on: value-adding process, resource processes HRM, ICT, infrastructure, work environment, managerial processes. In this way the knowledge management becomes integral part of the QMS of the institution. maria.dologova@statistics.sk, jandangi@gmail.com Implementation of an ISO Based Plan-Do-Check-Act Cycle in Statistics Lithuania Daiva Jurelevičienė, Bronislava Kaminskienė, Jūratė Kelmelytė, Laura Lukšaitė - Statistics Lithuania In 2007, the consistent work of specialists of Statistics Lithuania was awarded with a certification on the conformity with ISO 9001:2000 requirements. A Quality Management System implemented at Statistics Lithuania is a systematic approach towards quality management, based on process management, allowing effective organization of the institution s activity, well-balanced distribution of resources, expeditious reaction to the needs and expectations of users, respon- 22

5 dents and other institutions managing official statistics. The decision to use ISO 9000:2001 as a framework for quality management was taken in The main reasons for that were based on the following aspects embedded in the standard: Clearly defined and documented responsibilities and procedures; Structured approach towards the control and improvement of the processes; Process control based on factual and up-to-date information; Good framework for practical implementation of the plan-do-check-act cycle (hereinafter referred to as the PDCA cycle ). The aim of the paper is to present the course of the implementation of the PDCA cycle in Statistics Lithuania, which is based on ISO 9001:2000. The paper also deals with experience of Statistics Lithuania in integrating provisions of ISO 9001:2000 with the ESS quality framework. It describes the main methods and tools used by Statistics Lithuania for quality assessment and continuous improvement: audits of the Quality Management System, self-assessments, quality indicators, management of nonconformities and preventive actions, staff opinion and users satisfaction surveys, etc. Quality control in the core process of statistical production and dissemination is explored in more detail. The paper presents how different quality monitoring methods and tools were integrated and used in practice, how they support and supplement each other. One of the important requirements of ISO 9001:2000, implementing the PDCA cycle, is related with the control of nonconforming product and eliminating causes of nonconformities (i.e. corrective actions), as well as eliminating causes of potential nonconformities (i.e. preventive actions). This provision is extremely important not only in terms of continuous improvement and risk control, but also is strongly related with the development of quality culture in the institution. Also it goes along with strengthening loyalty of the staff and the improvement of working climate. Thus, the paper presents how the implementation of these procedures affected the daily work of the institution. daiva.jureleviciene@stat.gov.lt,bronislava.kaminskiene@stat.gov.lt,jurate.kelmelyte@statgov.lt, laura.luksaite@stat.gov.lt PLAN-DO-CHECK-ACT CYCLE ISO 9001:2000 ESS QUALITY FRAMEWORK Associated papers Measuring the Quality of Methods Used for the Measurement of Government Activity Emily Carless - Office for National Statistics, UK In any new or evolving area of statistics it is important to have a transparent and robust system for deciding which methods to use. The UK Centre for the Measurement of Government Activity, UKCeMGA, was set up in the Office for National Statistics in 2005 to drive forward improvements to the measures of UK Government Activity in the National Accounts. The need for a tool to improve the effectiveness of deciding which methods to use led to the UKCeMGA developing a framework for measuring the quality of existing and proposed new methods. The framework builds on international work on statistical (or output) quality and that on process quality. The framework is based on the six European Statistical Service Dimensions of Quality developed by Eurostat, namely, Relevance, Accuracy, Timeliness, Accessibility and Clarity, Comparability and Coherence. Six attributes of process quality have also been identified by a Eurostat working group, namely, efficiency, effectiveness, robustness, flexibility, transparency and integration. These are combined to provide a framework that will give the producers of the statistics the information they need to decide on their methods, and users the infor- OUTPUT QUALITY PROCESS QUALITY QUALITY OF METHODS 23

6 mation they need to understand why a particular method was chosen, and what its advantages and disadvantages are compared with some alternatives. The framework will also be used to prioritise the programme for improving existing methods and developing new ones based on identifying the areas of current methodology that appear to be of the lowest quality. This paper will outline the framework that UKCeMGA developed, give an example of how it has been used in practice, and offer suggestions about its wider applicability in the production of official statistics. QMS Implementation from the Basic Form to the Continual Improvement QUALITY MANAGEMENT SYSTEM CUSTOMER ORIENTATION CONTINUAL IMPROVEMENT Mária Dologová - Statistical Office of the Slovak Republic Ján Dolog - EOQ Senior Consultant for Quality Management Systems, Slovakia Implementation of the quality management system (QMS) always presents a significant change in the organisation the NSI is no exception. To avoid, from the long-term point of view, the unnecessary effort in the implementation and improvement of the system, it is important right from the beginning to set up a system that can easily be improved further as the development of the customer oriented institution requires it and that can easily integrate requirements formulated by other frameworks in the ESS. This approach was used in the implementation of the QMS in the Statistical Office of the Slovak Republic (SOSR). Decision about the QMS The SOSR decision to implement the QMS was based on the need to provide high quality value to its stakeholders in a very demanding environment. The system chosen had to be simple, process oriented, had to ensure repeatability in meeting stakeholders requirements and to facilitate costs optimisation. Moreover, it had to allow any further enhancement, incl. integration of requirements of other systems or frameworks, e.g. Code of Practice, Eurostat Quality Assurance Framework, EFQM, etc. For this reason the SOSR decided to implement the QMS according to ISO 9001:2000 standards, which is based on the principles necessary to achieve the objectives: customer / user focus, leadership, process approach, system approach to management, factual approach to decision making, continual improvement. One of the most decisive factors that influenced the choice of the system was the approach to one of the main principles to the process approach: while the QMS according to ISO standards mandatory introduces the process approach, other (higher) systems assume that the organization already has it. Setting up the QMS Within the system the main processes were identified. The value-adding processes create the core these are the processes that directly add value to stakeholders (here statistical processes). Other processes identified create the necessary support. These are managerial processes that cover the whole managerial process from strategy to operation and supporting processes that ensure the necessary resources for all processes. The process description covers items necessary for managing the process input, output, activities, resources needed in the process, regulations as well as the managerial tools: performance indicators, process parameters, the way of monitoring, measuring and improvements. The concrete wording depends on the development direction of the institution reflected in the strategic statements, quality policy and strategic / quality objectives. The decomposition of the process goes down to activity level. 24

7 The system is monitored by satisfaction surveys, self-assessments, internal audits and external audits (conducted by the certification body) and by management review. The audit orientation reflects the medium and long-term objectives of the institution. The system approach ensures that the QMS is also able to cover items of the Eurostat Quality Assurance Framework. Continual improvement of the QMS The continual improvement is ensured on the system level and on the operational and tactical level. On the system level via the implementation of the new or the enhancement of the existing pillars of the system - as required by the challenging environment reflected in the long-term development strategy of the institution. On the operational and tactical level via good established and managed processes by the continual improvement of the process composition and the process performance. The common tools used here are the strategic statements, quality policy, strategic / quality objectives, audit results, corrective / preventive actions, management review results. As the experience shows, the optimal QMS is tailor-made it may consist of the pillars taken as inspiration from the existing models (EFQM, CAF, ISO 9004, etc.) and of the own pillars. They create one integral QMS implemented to meet requirements of the institution. maria.dologova@statistics.sk, jandangi@gmail.com 25