Quality Management in Korean National Statistical Systems Focused on Quality Assessment

Transcription

1 Quality Management in Korean National Statistical Systems Focused on Quality Assessment Sung H. Park Department of Statistics, Seoul National University, Seoul, Korea ABSTRACT First of all, the current status of Korean official statistics is reviewed, and then some basic concepts of statistics quality management in Korea are explained. The core of statistics quality management is the quality assessment system, which is newly developed in Korean statistical systems. Secondly, the guidelines of quality assessment system in Korea are explained in detail. The 6 assessment areas are assessment of statistics production environment, adequacy assessment of statistics production procedures, accuracy assessment of data collection, perfection assessment of published data, assessment of user satisfaction, and assessment of improvement effort for statistics quality. Lastly, a way of summarize the assessment results is presented. I. INTRODUCTION With a growing demand of accurate and timely informations in this knowledge-based information society, the decision-makers at all levels of organizations including government departments and business enterprises have become increasingly dependent on statistics for fundamental data for decision making. In this regards, the quality of statistics have become more and more important, and the production organizations of statistics have been aware of the importance of statistics quality. It is known that the first effort for a guideline for statistics quality was made by Statistics Canada in 1985, which was named 'Statistics Canada Quality Guidelines'. Since then, many national statistical offices and international organizations have participated in this effort to make guidelines for statistics quality. For instance, Eurostat began 'Quality Assessment System' from 1994, and IMF announced 'Data Dissemination Standards' in 1996 to enhance the statistics quality. Also OECD began 'Quality Management System for Statistics' from 2001 for quality assurance of official statistics. The Korean National Statistical Office (KNSO) realized the importance of statistics quality management since the mid-1990s. In 1999 the KNSO introduced Statistical Inspector System by which the Quality Assessment Team (QAT) under the Statistics Planning Division was established. The inspectors in the QAT are to take a neutral stance in review and assessing the quality of individual data using the Quality Checklist with quality indicators and making a report on the outcome of review and assessment which are submitted to the KNSO. Later, the QAT was promoted to the Quality Management Department. Despite the globally common effort toward the improvement of data quality, the international community finds it hard to define the theoretical aspects of statistical quality assessment, as the expected level of statistical quality varies depending upon the user group and the way to meet the quality demands differs from country to country according to the statistical development phase or statistic culture of a country. This paper intends to describe the guidelines for statistics quality management in Korean statistical systems. II. REVIEW OF KOREAN OFFICIAL STATISTICS The statistical system in Korea is decentralized like U.S.A., France and Japan. There are total 500 national statistics produced currently which are controlled by Statistics Law. Among them, the KNSO produces 53 fundamental national statistics, and the other individual statistical agencies produce the remaining statistics as shown in Table 1. Besides these 500 statistics, there are many statistics produced by civilian organizations, which are not controlled by Statistics Law. 1

2 Table 1: Classification of 500 National Statistics (as of ) Organization (numbers) KNSO (1) Central government agencies (29) Local government agencies (16) Civilian organizations (75) Function Production of fundamental national statistics, Statistical mediation of individual agencies Production of responsible statistics Production of statistics resulted from local administration Production of statistics related to individual activities Produced statistics Total 53: Population Census Economically Active Population Survey Social Statistics Survey Consumer Price Survey, etc. Total 203: Total 98: Total 146: National accounts Under the decentralized system, the possibility is that statistical data can be duplicated, which causes confusion to users of statistical data. By Statistics Law, the KNSO is supposed to play the mediating role for individual agencies: in case the statistics that an individual agency intends to take is similar to any existing statistical data, the KNSO prohibits the agency from working on the statistics. Also in the decentralized system, the statistics users have to take the trouble to visit each and every database of the statistical agencies. In order to overcome the inconvenience of users, the KNSO developed STAT-Korea, a portal internet site for statistical data in STAT-Korea has since provided one-stop service for users of statistical data, by linking the KNSO database to the databases of the other statistical agencies. It is believed that the national statistical system in Korea has the following problems, which should be overcome for a better national statistical system. (1) Development of new statistics is rather slow. The society is rapidly changing and there has been a growing demand for understanding the changing society. In particular, new statistics are needed in the areas of biotechnology, environment, foreign investment, knowledge asset, leasure, local government, and so on. (2) Quality management is weak in the production process of statistics. The quality management system is not much effective for the most government agencies and civilian organizations which produce national statistics. The production process usually consists of 4 stages; planning, survey, analysis and publication. It seems the planning/analysis stages are particularly weak. The manpower of statistical planning/analysis in Korea is roughly 1/10 compared with those of developed countries as shown in Table 2. Table 2. Manpower of Planning/Analysis in Central Statistical Agencies (2004) country Korea U.S.A. Canada Netherlands Australia Denmark number of persons per one million people (3) Statistical production agencies are not active in sharing and utilization of statistical data. This is partly because of the decentralized system of statistics, and partly because of poor cooperation attitude between production organizations. This problem is strong in the areas of tax, employment insurance, public health, house construction, small areas, and so on, which are related to personal informations. 2

3 III. BASIC ASPECTS OF STATISTICS QUALITY MANAGEMENT 1. Definitions of terminology (1) Statistics quality The term 'quality' is one of the words most widely used in the modern society. And yet, the word has no single fixed concept and can be used to mean different things in different situations. The most popular definitions of quality are the following. Juran and Gryna (1980): "fitness for use" ISO 8402 (1986): "the totality of features and characteristics of a product or services that bear on its ability to satisfy stated or implied needs" However, recently the user satisfaction is the key word in quality management. In this respect, the definition of statistics quality used in the KNSO (2006) is as follows. Statistics quality is "the totality of characteristics of a statistic for user satisfaction based on fitness for use". (2) Quality management Quality management (QM) is the system of management to maintain the desired quality. Since the PDCA (plan, do, check, act) cycle is often used in QM, KNSO (2006) defines QM as follows. Quality management is "all kinds of activities using the PDCA cycle for securing and maintaining the user required quality in systematic and economical ways through quality planning and quality improvement efforts". (3) Quality dimensions Quality has many dimensions, all of which should be determined by the uses made of the data by the key users. The seven dimensions of quality which are adopted by the KNSO for quality assessment are: * accuracy: closeness to true value * timeliness: frequently & on time * relevance: meet users' needs * accessibility: easily accessible & assistance in using statistics * comparability: across time & space * efficiency: efficient ways & easy burden * serviceability: easily serviceable for users Table 3 shows the comparison of quality dimensions among several organizations. Table 3. Comparison of quality dimensions quality Nether- Korea Canada dimension lands Eurostat IMF OECD Sweden integrity * methodological soundness * relevance * * * * * accuracy * * * * * * * accessibility * * * * * * timeliness * * * * * * serviceability * * coherence * * * * comparability * * * interpretability * * * efficiency * * reliability * without too much burden * 3

4 (4) Quality indicators For each of quality dimensions, there are several or more quality indicators which measure the quality dimension. There are quality indicators in the quality check list. (5) Statistics quality management system Statistics quality management system is the system which combines all methods to produce statistics in order to fulfill the quality dimensions economically as way as well as to satisfy the users' requirements. (6) Statistics quality assessment system Statistics quality assessment system is the overall assessment system for statistics quality from statistics planning to publication through assessing statistics production environment, adequacy of statistics production procedure, accuracy of survey, soundness of publication data, degree of satisfaction of statistics users, improvement effort of statistics quality, and so on. 2. Eight principles for successful statistics quality management In order to implement a successful statistics quality management in any organization, the following 8 principles should be kept. (1) User oriented (2) leadership (3) full participation of all parties concerned (4) process oriented approach (5) systematic approach for management (6) continuous improvement (7) fact and data-driven approach for decision-making (8) involvement of suppliers The last principle means that, if a partial work of statistics production is given to an outside source (a supplier), the supplier should be involved in the statistics quality management system of the statistics production organization. 3. A current effort of statistics quality management To assess the quality for all 500 national statistics, a 3-year quality assessment project is under way. This year is the beginning year, in which relatively important 107 statistics are supposed to be assessed with budget of 1.6 million dollars. 12 assessment teams were selected last March, and they are assessing 107 statistics. The remaining statistics will be assessed next two years. The purpose of the assessment is to upgrade the quality levels of all national statistics, and finally is to improve the reliability of national statistics. Through this assessment, it is hoped that a sound statistics quality management system could be spread around to all statistics production agencies. IV. GUIDELINES OF QUALITY ASSESSMENT IN KOREAN STATISTICAL SYSTEMS A flow chart of statistics quality assessment process is recommended for all statistics production agencies in KNSO (2006). the flow chart is given in Figure 1. 4

5 Figure 1. Flow chart of statistics quality assessment process Selection of statistics for quality assessment? Assessment area 1. Assessment of statistics production environment 2. Adequacy assessment of statistics production procedures Assessment methods * current status of production * survey on recognition of the persons in charge * self-assessment by quality check-list * outside assessment by quality review members * check of survey contents 3. Accuracy assessment of data collection through telephone re-interview * error check of 4. Perfection assessment of statistical periodicals published data *? check of users' convenience * check of the using status of statistics 5. Assessment of user * measure of satisfaction satisfaction * understanding of users' requirements 6. Assessment of improvement effort for statistics quality * check on quality improvement plan * assessment of results Quality dimensions all (7 dimensions) All Accuracy Accessibility comparability serviceability All All? Completion of quality assessment and result analysis? Preparation of overall assessment report V. DETAILS OF QUALITY ASSESSMENT 1. Assessment of statistics production environment An important factor which influences the statistics quality is the statistics production environment such as top leadership and manpower resource. A sound statistics production environment is a basic infra structure for the statistics quality management. The way to assess the statistics production environment is shown in Table 4 as a flowchart. 5

6 Table 4. Overall flow for assessment of statistics production environment Assessment flow 1. Set-up of assessment plan for statistics production environment 2. Preparation/completion of inquiry report on statistics production environment 3. Interview with responsible persons in charge of statistics production 4. Summary of inquiry results and assessment 5. Preparation of assessment report and feedback Detailed contents * set-up of detailed assessment plan * opinion feedback from the persons in charge of statistics The inquiry report contains the basic environment, production conditions, recognition of the responsible persons for the applicable statistics. Interview with the responsible persons who provide the inquiry report Summary of inquiry results and utilization of assessed scores * Opinion check from the responsible persons about the inquiry results * Feedback for improvement information 2. Adequacy assessment of production procedures The adequacy assessment of production procedures is done through self-assessment as well as outside assessment. The self-assessment can be done by completing the checklist from the responsible persons. The outside assessment can be done by re-assessment of the completed checklist by the responsible persons. The outside assessors are also supposed to propose some measures for quality improvement of the concerned statistics. The overall work flow is shown in Table 5. The 5 point scale for evaluation is based on the following criterion. Very excellent: above 4.5 points in average Excellent: below 4.5 points Normal: below 4.0 Poor: below 3.5 Very poor: below 3.0 Table 5. Overall flow for adequacy assessment Assessment flow 1. Set-up of assessment plan for adequacy of production procedures 2. Check of statistics quality indicators 3. Appointment of statistics quality Review members 4. Preparation of quality assessment Form 5. Analysis of assessment results and preparation of report 6. Feedback Detailed contents * Set-up of detailed assessment plan * Announcement of the plan * Reflection of opinions of the persons in charge of the assessed statistics * Data collection and analysis for selection of statistics quality indicators * Benchmarking of other countries, and literature survey * Recommendation of review experts from the to-be-assessed department * Organization of the statistics quality review board * Preparation of questionnaire type assessment form * Implementation of self-assessment and outside assessment by this form * Result analysis of the self-assessment and outside assessment * Evaluation of the results by 5 point scale * Preparation of report * Transmission of the feedback report to the assessed department * Request of quality improvement plan 6

7 There are total 35 quality indicators for adequacy assessment of production procedures. They are distributed as shown in Table 6. The average score to judge the adequacy of production procedures is computed by the following formula. Table 6. Quality assessment indicators for adequacy of production procedures procedures/ dimensions total 1. survey planning population/s survey ample design 4. interviewer training 5. survey 6. data processing 7. use data Total accuracy timeliness relevance 2 2 accessibility comparability 2 2 of Average score =( 5*A + 4*B + 3*C + 2*D + E ) / (A + B + C + D + E) Where A = number of indicators with 'very excellent', B = number of indicators with 'excellent', C = number of indicators with 'normal', D = number of indicators with 'poor', E = number of indicators with 'very poor'. 3. Accuracy assessment of data collection The accuracy of a statistic depends on the data collected at the actual spot where a set of data is accumulated through interview. This means that the accuracy of a statistic is dependent on the interview process as well as on the efficiency of the data collection system. The accuracy assessment of data collection can be mainly done by re-interview of the respondents. The re-interview is usually done by telephone. The major goal of this assessment is to find the source of non-sampling error during the interview process, and to improve the survey data quality. The overall flow of the re-interview process for accuracy assessment of data collection at the actual spot is given in Table 7. Table 7: Overall flow of re-interview for accuracy assessment of data collection 1. Assessment planning 2. Sample selection (sample size: about 10% of respondents or at least 100) 3. Preparation of telephone scenario 4. Appointment of telephone interviewers and their training 5. Implementation of telephone re-interview 6. Confirmation of telephone re-interview and summary results 7. Preparation of report and its feedback 4. Perfection assessment of published data Even though a statistic is without error in the production process, if there are some errors in the publication process, there is no guarantee that the published data are in good quality. There are so many published materials and survey reports. However, the perfection assessment of the published materials is rather weak in Korea. the purposes of this perfection assessment are two. The first is to measure the types of error and their frequencies, and then to prevent them to happen again. The second is to improve the serviceability of the published statistics by checking whether the needed fundamental information is provided to users. The overall flow of perfection assessment is as follows in Table 8. 7

8 Table 8: Overall flow of perfection assessment of published data 1. Assessment planning for error assessment and user serviceability 2. Appointment of assessors for published data and their training 3. Implementation of perfection assessment 4. Confirmation of the assessed contents and summary results 5. Preparation of result report and feedback 5. Assessment of user satisfaction The degree of user satisfaction is based on the fitness for use. If the data do not fit well to the requirements of the users, the user satisfaction would be low. The purpose of the assessment of user satisfaction is to find out the projects for quality improvement. The overall flow for assessment of user satisfaction can be found in Table 9. Table 9: Overall flow of assessment of user satisfaction 1. Assessment planning for user satisfaction 2. Preparation of users' list, and selection of a sample of at least 100 users 3. Design of survey questionnaires 4. Implementation of survey 5. Data input, summary and analysis 6. Preparation of report and feedback To measure the user satisfaction, the following 5 scale is recommended. 5 point: very satisfied 4 point: satisfied 3 point: normal 2 point: not satisfied 1 point: not at all satisfied If there are k items (questions), the degree of total satisfaction (S) can be computed by the following formula. S = wi * Si where Si is the satisfaction of the ith item and wi is the importance weight for the ith item. 6. Assessment of quality improvement effort Generally speaking In quality management, continuous improvement effort is essential. It is also true in statistics quality management. In order to improve quality continuously, it is necessary to find quality problems, and make them improvement projects, and work on them continuously. The overall flow of assessment for improvement effort for statistics quality is in Table 10. Table 10: Overall flow of assessment of improvement effort for statistics quality 1. Quality improvement planning 2. Data request and data collection 3. Implementation of assessment 4. Preparation of report and feedback 7. How to summarize the assessment? After assessing the 6 areas mentioned above, it is advisable to make a summary table as shown in Table 11 for an illustration. The result of each area can be evaluated in 5 point scales as follows. 5 point: very good 4 point; good 8

9 3 point: normal 2 point: poor 1 point: very poor In Table 11, we assume that each of 6 areas has the same importance weight. However, we may have some weight to each area, and then we can find a weighted average score, if we want. Table 11: Summary result of quality assessment Statistics Statistic A Statistic B Overall evaluation average score 3.5 (between 2.8 (near to normal normal and good) from poor) Statistics production environment 4 (good) 3 (normal) Adequacy of production procedures 3 (normal) 2 (poor) Accuracy of data collection 5 (very good) 1 (very poor) Perfection of published data 2 (poor) 4 (good) Statistics user satisfaction 3 (normal) 4 (good) Quality improvement effort 4 (good) 3 (normal) VI. CONCLUDING REMARKS Each country may take a different approach to assessment areas, assessment methodology, and selection of assessment indicators. These elements are associated with the level of statistical development of a country, and the statistical culture and environment. However, for a successful quality assessment, the above mentioned 6 areas of assessment should be required. The usefulness of a statistics quality assessment system can be proven only when the assessment results are constantly fed back to the survey process and, thereby, improve the data quality. REFERENCES 1. Korean National Statistical Office (2006), Statistics Quality Management Handbook. 2. Korean National Statistical Office & International Monetary Fund (2000), The Proceedings of Statistical Quality Seminar 2000, 6-8, December 2000, Jeju, Korea. 3. Park Sung H. and Park Jin W. (2003), A study on Standardized Manual for Statistics Quality Management, Funded by Korean National Statistical Office. 9