South African Statistical Quality Assessment Framework 30 August 2011 National Statistics System Division 1
Overview Purpose Background Definition Statistics Act South African Statistical Assessment Framework (SASQAF) Purpose of the framework Structure of the framework Definition of statistical quality Dimensions of Quality Practical Application Conclusion 2
Purpose To explain the current practice in the assessment of the quality of statistical products within the broader NSS through application of South African statistical quality Assessment Framework (SASQAF) 3
Background The statistical landscape is characterized by three gaps Capacity gap Quality gap Information gap To address the Quality Gap the SG has gazetted through parliament the SASQAF on the 23 rd September 2009. SASQAF quality requirements must be met by all statistics producers in the NSS to qualify as Official Statistics. 4
Definition Official statistics definition is statutory see Statistics Act [No. 6 of 1999] Official statistics are statistics designated as official statistics by the Statistician-General within the provisions of the Statistics Act Practical criteria of official statistics Must be used in the public domain Are from organs of state and other agencies that are partners in the National Statistics System [NSS] Are sustainable Have met quality criteria as defined by the Statistician-General [SASQAF] National statistics definition is implicitly statutory National statistics are statistics not designated as official statistics by the Statistician-General 5
The Statistics Act No. 6 of 1999 Stats SA is governed by the Statistics Act (No. 6 of 1999) The Stats Act covers Official Statistics and Other statistics Implies, Stats Act covers all statistics produced that informs policy, planning and monitoring of government performance. Section 14(7) of the Act, empowers the Statistician-General to designate as official statistics any statistics or class of statistics produced by Stats SA or any organ of state Required that a rational, sustainable and transparent framework for assessing the quality of those statistics being developed South African Statistical Quality Assessment Framework (SASQAF) has been developed for this purpose 6
SASQAF - Based on international best practice World Bank EuroStat South Africa Canada UK Prerequisites of quality Prerequisites of quality Relevance Relevance Relevance Relevance Accuracy & Reliability Accuracy Accuracy Accuracy Accuracy Accessibility Timeliness & Punctuality Accessibility & clarity Timeliness Timeliness Timeliness & Punctuality Accessibility Accessibility Accessibility & clarity Coherence Coherence & Comparability Coherence Coherence Methodological Soundness Integrity Interpretability Methodological Soundness Integrity Interpretability Serviceability Comparability Comparability 7
South African Quality Assessment Framework (SASQAF) Developed by Stats SA with the help of international consultants Based on the Data Quality Assessment Framework (DQAF) of the International Monetary Fund (IMF) It covers - all activities related to the statistical value chain for sample surveys and administrative records 8
Purpose of the framework Provides a structure for the assessment of statistical products based on 1. Dimensions of quality 2. Indicators, and 3. benchmarks Used in various contexts Reviews within NSS Self-assessment by data-producing agencies Assessment by data users Assessment by international agencies e. g. IMF 9
Structure of the framework Dimension Prerequisites of Quality Relevance Accuracy Timeliness Accessibility Interpretability Comparability and Coherence Methodological soundness Integrity Quality Indicators 8 indicators 5 indicators 7 indicators 4 indicators 13 indicators 3 indicators 6 indicators 6 indicators 6 indicators 58 indicators 10
Structure of the framework Each of the 8 quality dimensions consists of number of indicators Within the indicators a number of benchmarks are identified relating to a 4-point scale; Quality statistics (4) Acceptable statistics (3) Questionable statistics (2) Poor statistics (1) 11
Quality dimension Description 0. Prerequisites of quality Refers to the institutional and organisational conditions that have an impact on data quality. Key components Legal and institutional environment (including Memoranda of Understanding (MoUs) or Service Level Agreements (SLAs) Privacy and confidentiality Resources are commensurate with the needs of statistical programmes Quality is the cornerstone of statistical work Structure of the framework Indicator The responsibility for producing statistics is clearly specified. Standards and policies are in place to promote consistency of methods and results. Data sharing and coordination among data-producing agencies is clearly specified and adhered to. Quality Statistics Acceptable Statistics Assessment Levels Questionable Statistics Poor Statistics Level 4 Level 3 Level 2 Level 1 The responsibility for producing statistics is explicitly specified through a legal framework. All standards and policies are in place to promote consistency of methods and results, and are adhered to. Data sharing and coordination among dataproducing agencies is explicitly specified through a legal framework. The responsibility for producing statistics is specified through a legal framework. The majority of standards are in place to promote consistency of methods and results. Data sharing and coordination among dataproducing agencies is specified through a legal framework. The responsibility for producing statistics is implied through a legal framework. Some standards are in place to promote consistency of methods and results. Data sharing and coordination among dataproducing agencies is implied through a legal framework. The responsibility for producing statistics is not specified. No standards are in place to promote consistency of methods and results. Data sharing and coordination among dataproducing agencies is not specified. 12
Definition of Statistical Quality Quality of a statistical product is defined in terms of fitness for use Quality can be decomposed into a number of quality dimensions. SASQAF identifies 8 dimensions of quality Each dimension has associated quality indicators, standards and benchmarks. 13
Quality Dimensions: Review SASQAF identifies 8 quality dimensions Free from political interference: Adherence to objectivity, professionalism, transparency, ethical standards Coherence Integrity Relevance SASQAF (data quality) Meeting real needs of clients Methodological soundness Sound methodologies: International standards and guidelines - good practice Agreed practices Dataset-specific Harmonisation of different info within broad analytical and temporal framework Interpretability Availability of supplementary info and metadata Accessibility Ease of obtaining info from agency Timeliness Info available at desired reference point Accuracy Correctly describes phenomena it is designed to measure 14
SASQAF & Generic Statistical Value Chain Quality Management & Metadata Management 1 Need 2 Design 3 Build 4 Collect 5 Process 6 Analyze 7 Disseminate 8 Archive 9 Audit 1.1 Determine need for information 1.2 Consult & confirm information requirements 1.3 Establish output objectives 1.4 Check data availability 1.5 Prepare business case 2.1 Outputs 2.2 Frame and sample methodology 2.3 Tabulation Plan / Variables 2.4 Data collection 2.5 Statistical processing methodology 2.6 Define archive rules 2.7 Processing systems and workflow 2.8 Detailed project plan 3.1 Data collection instrument 3.2 Process components 3.3 Configure workflows 3.4 Test end-toend 3.5 Finalise production systems 3.6 Draw sample 4.1 Set up collection 4.2 Run collection 5.1 Standardize 5.1 Classify and code 5.2 Load data into processing environment 5.3 Integrate data 5.4 Edit and impute 5.5 Derive new variables 5.6 Calculate weights 6.1 Acquire ancillary information 6.2 Calculate aggregates 6.3 Prepare draft outputs 6.4 Validate 6.5 Describe and explain 6.6 Disclosure control & Anonymise 6.7 Finalize outputs for disseminatio n 7.1 Update output systems 7.2 Produce products 7.3 Produce Quality Statement. 7.4 Manage release of products 7.5 Market and promote products 7.6 Manage customer queries 8.1 Manage archive repository 8.2 Preserve data and associated metadata 8.3 Dispose of data and associated metadata 9.1 Gather inputs for audit 9.2 Prepare audit report 9.3 Quality plan 15
1. Need 1.1 Determine need for information 1.2 Consult & confirm information requirements 1.3 Establish output objectives 1.4 Check data availability 1.5 Prepare business case initial investigation identification of what statistics are needed & for what purpose. what is needed of the statistics. Practice amongst other organisations, & methods used by those organisations. consulting with the stakeholders (detail need). user needs are required (why, what, when, how). Subsequent iterations of this phase to ascertain whether previously identified needs have changed (critical to this subprocess). identifies the statistical outputs required. ensure output to meet the user needs identified. Agree on suitability of proposed outputs. Agree on quality measures with users. NOTE: This will include reaching agreement on which SASQAF quality indicators it is expected to report on. The survey area negotiates this with M&E Check if current data sources could meet user requirements. Conditions for availability? Restrictions on their use? Research potential administrative data sources and their methodologies. Determine suitability of admin sources. Develop & prepare strategy for filling any Document the findings of previous subprocesses. formulate business case to get approval to implement. These Include: Description of the As- Is business process (if it already exists), highlight any inefficiencies & issues to be addressed, Proposed To-Be solution, Statistical business process to be developed, Costs and benefits External constraints, & Budget allocation. remaining gaps 16
Process Audit 17
Process Audit 18
19
20
Export Ready Results 21
Action Plan 22
Adherence to SASQAF will Conclusion encourage compliance to the agreed standards, procedures and guidelines resulting in improvement of quality, closing the quality gap ensure that more statistics are certified as official closing the supply gap assist the users in assessing the quality of data and products promoting transparency ensure that all published products include statements about data quality (quality declaration) informing users about the quality of data and products ensure that more statistical products produced in the NSS are declared as fit for use 23
Thank you 24