National Quality Assurance Frameworks Mary Jane Holupka September 2013 1
Presentation 1.1 - OUTLINE Understanding Quality BASIC CONCEPTS - Quality and Dimensions of Quality - Quality Assessment - Quality Assurance 2
Better Quality?? 3
What is QUALITY? - A rather vague concept (not many synonyms for it), has different meanings depending upon the context. -In the NSO context, QUALITY is defined as FITNESS FOR USE, in terms of user needs - - how well the agencies products meet user needs, whether they are fit for use or fit for the purpose for which they are to be used. -The NSO s product is the INFORMATION it disseminates (facts to be used for decision-making by governments, businesses, institutions, the public); the focus here is on Information Quality. **NSO=national statistical organization = CSO, NSI, etc. 4
What is QUALITY? FITNESS FOR USE This definition is broader than in the past when quality was equated with accuracy. Now it s recognized that there are other important dimensions. Can data be said to be of good quality when: ACCURATE - - but produced too late to be used?? ACCURATE - - but can t be found, accessed, or totally understood?? ACCURATE - - but conflict with other data?? Thus QUALITY needs to be looked at as a multi-faceted, multi-dimensional concept Its definition is a relative one. Depends upon the intended uses of the outputs. The uses will vary across different groups of users. 5
QUALITY DIMENSIONS (COMPONENTS) The dimensions or components to be considered when assessing the quality of data outputs (i.e. product quality) are, according to the NQAF: 1. Relevance 2. Accuracy and reliability 3. Timeliness and punctuality 4. Accessibility and clarity (covered in NQAF14) (covered in NQAF15) (covered in NQAF16) (covered in NQAF17) 5. Coherence and comparability (covered in NQAF18) These are not the only possible dimensions. Other organizations and countries use these and/or: interpretability; credibility; integrity; methodological soundness; serviceability. -The dimensions are overlapping and interrelated; data quality should be seen as a balance of a number of different dimensions or components (not just accuracy!), and every dimension needs to be adequately managed if information is to be fit for use. Failure in any one dimension will impair the usefulness of the information. -- COSTS also need to be considered. 6
NQAF QUALITY DIMENSIONS Timeliness and punctuality Information is made available soon after the end of the reference period, and when it was promised Coherence and comparability Data can successfully be brought together with/related to other statistical data; Data can be compared across countries, over time Relevance Data quality Ease of obtaining Info; availability of supplementary info and metadata Accessibility and Clarity Meeting needs of users Accuracy and reliability Correctly describes phenomena it is designed to measure 7
QUALITY DIMENSIONS (COMPONENTS) 1. Relevance The degree to which statistical outputs meet current and potential user needs. - Do the data meet the current and/or potential requirements of users, clients, stakeholders, and the audience? - Are the statistics that are needed produced? - Are the statistics that are produced needed? - To what extent do the concepts used (definitions, classifications etc.) reflect user needs? 8
QUALITY DIMENSIONS (COMPONENTS) 2. Accuracy the degree to which the information correctly describes the phenomena it was designed to measure, i.e. the degree of closeness of estimates to true values. -How close are the statistical estimates are to the true values? 2. Reliability closeness of the initial estimated value to the subsequent estimated value. - It concerns whether the statistics consistently over time measure the reality that they are designed to represent. 9
QUALITY DIMENSIONS (COMPONENTS) 3. Timeliness length of time between data availability and the event or phenomenon they describe. -How long is the time between data being made available and the event or phenomenon they describe? (Data release vs the end of the reference period). -Typically involved in a trade-off against accuracy. -Timeliness of information will influence its relevance. 3. Punctuality refers to whether data are delivered on the dates promised, advertised or announced. -What was the time lag between the target date when they should have been released/were promised and the actual release date of data? 10
QUALITY DIMENSIONS (COMPONENTS) 4. Accessibility the ease and conditions under which statistical information can be obtained. -where to find the data; how to order; what s the pricing policy; what formats are available (paper, electronic files, CD-ROM, Internet ). - making them available (on the Internet or in a book) is important, but accessibility also entails bringing data to users in an understandable way, opening a dialogue with those users, and ensuring that their information needs are met. 11
QUALITY DIMENSIONS (COMPONENTS) 4. Clarity the extent to which easily understandable metadata are available, where these metadata are necessary to give a full understanding of statistical data. (Sometimes referred to as interpretability ). - accompanied by sufficient and appropriate metadata? - graphs or maps to add value to the presentation of the data? -information on data quality? 12
QUALITY DIMENSIONS (COMPONENTS) 5. Comparability degree to which data can be compared over time, geographic areas or other relevant domains. -Over time can data from different points in time be compared? Across countries and/or regions? 5. Coherence adequacy of statistics to be combined in different ways and for various uses. - To what degree can the data be successfully brought together with other statistical information? Internal, cross-domain coherence; provisional vs final. 13
Quality dimensions used in several international organizations 14
Quality dimensions used in some countries Canada South Africa UK Prerequisites of quality Relevance Relevance Relevance Accuracy Accuracy Accuracy Timeliness Timeliness Timeliness & Punctuality Accessibility Accessibility Accessibility & clarity Coherence Coherence & Comparability Coherence Comparability Interpretability Interpretability Methodological Soundness Integrity 15
Which is the Best Quality? It depends on user s needs! 16
Quality is not just about outputs Inputs Processes Outputs To have good outputs we need to have good inputs and processes, so we need to think about the quality of these as well - and the quality of the organization responsible for the processes (institutional environment), and the quality of the NSS too. 17
Quality assurance defined Definition: All the planned and systematic activities implemented that can be demonstrated to provide confidence that the processes will fulfil the requirements for the statistical output. Source: SDMX (2009) - Anticipating and avoiding problems with the goal of preventing, reducing or limiting the occurrence of errors in a survey; getting it right the first time. It s an organization s guarantee that the product or service it offers meets the accepted quality standards. 18
Quality assessment defined Definition: Quality assessment is a part of quality assurance that focuses on assessing (evaluating determining) the extent to which quality requirements have been fulfilled. Context: "Quality assessment" contains the overall assessment of data quality, based on standard quality criteria. This may include the result of a scoring or grading process for quality. Scoring may be quantitative or qualitative. Source: Based on ISO definitions: ISO 9000/2005: Quality Management and Quality Assurance Vocabulary Hyperlinks: Above taken from ESS Quality Glossary 2010 19
Quality control defined Definition: Subset of quality assurance (QA) process, it comprises activities employed in detection and measurement of the variability in the characteristics of output attributable to the production system, and includes corrective responses. (Source: Economic Commission for Europe of the United Nations (UNECE), The Knowledge Base on Statistical Data Editing, Online glossary developed by the UNECE Data Editing Group, 2000). Quality control is directed at what can be measured and judged acceptable or unacceptable. A technique to check quality against a set standard or specification. 20
Quality assurance summary A system of coordinated methods and tools to ensure a sustainable level of quality of processes and outputs, where: product/ output quality requirements are explicitly documented processes are defined and made known to all staff the correct implementation of the processes is monitored users are informed about the quality of the products and possible limitations a procedure is in place to guarantee that the necessary improvement measures are planned, implemented and evaluated 21
QUALITY Quality assurance Quality assessment Quality control Quality management Quality dimensions Quality components Quality frameworks Quality reports (user/produceroriented) Quality reviews Quality indicators Quality profiles (Eurostat) Quality declarations Quality statements Quality trade-offs Quality improvement actions Terminology QUALITY Product quality Process quality Template Framework NSO vs NSS Principles indicators - Eurostat Dimensions, elements and indicators - DQAF NQAF lines elements to be assured mechanisms http://unstats.un.org/unsd/dnss/docsnqaf/nqaf%20glossary.pdf 22
Terminology consult the NQAF website 23
Quality End of presentation 1.1 - Quality and Dimensions of Quality - Quality Assessment - Quality Assurance Thank you for your attention. 24