Quality principles and user needs of balance of payments statistics BALANCE OF PAYMENTS STATISTICS DATA COLLECTION AND COMPILATION Rome, 9-10 June, 2011 Giacomo Oddo
Overview Definition of quality and user needs The European Statistics Quality Framework Specific quality features for BoP statistics Concluding remarks
What is quality? Capability to deliver utility Adherence to desired characteristics Capability to fulfil user needs. What is statistics quality? Accurate and timely information on real world phenomena Non-distortionary and accountable Perceived as independent and objective.
Balance of payments statistics: who needs them and why? Internal Users: External Users: Executive Board Governing Council Economic Outlook Office Research Department International organisations Authorities Academia Financial analysts Media and public at large Monetary policy and macroeconomic analysis Structural analysis, economic forecast, statistics production page 1
Quality: from theoretical definition to practical implementation common standards and regulations IMF: Special Data Dissemination Standards (SDDS) The SDDS was established by the IMF to guide members ( ) in the provision of their economic and financial data to the public. SDDS are expected to enhance the availability of timely and comprehensive statistics and therefore contribute to the pursuit of sound macroeconomic policies ( ) and to the improved functioning of financial markets. Quality assessment Common standars, evaluation criteria, regulatory principles and best practices sharing. WHAT, WHEN, HOW must be disclosed to the public! http://www.dsbb.imf.org/pages/sdds/ctyctgbaselist.aspx?ctycode=ita&catcode=bop00
European regulation and common framework (Eurostat & ECB) Commission Regulation (EC) no. 1055/2008, regarding quality reporting for balance of payments statistics Mandatory quality report on annual basis six dimensions are identified as relevant: Timeliness Methodological soundness Stability Plausibility Consistency Accuracy and reliability Serviceability
Dimensions go across products, processes and producing institutions. Dimensions shall be encompassed in a coherent quality framework, i.e. in a set of quality principles. 1) Institutional Environment 2) Statistical Process 3) Statistical Output Producer duties (and needs) Quality of the Institution User needs Quality of the product
Quality elements of the Institutional Environment Independence and accountability of the official producing agency Mandate for data collection (legal framework) Impartiality and objectivity Statistical confidentiality Coordination and cooperation Resources endowment and efficient management http://www.dsbb.imf.org/pages/sdds/ctyctgbaselist.aspx?ctycode=ita&catcode=bop00
Independence and accountability EU Regulation emphasises the importance of the independence, integrity and accountability of the national statistical authorities. ECB legal framework of independence. Strategy, monitoring and reporting. After the Greek crisis, EC is trying to grant Eurostat additional rights to access to all the information required for the purposes of the data quality assessment and not just statistical information. Mandate for data collection Producing agencies need to have a clear mandate to collect statistical information from the economic agents or from other national authorities (espec at EU level). ECB is empowered to impose sanctions on reporting agents which fail to comply with their obligations.
Impartiality and objectivity Choice of data sources, methods and procedures used should be disclosed. Release calendars should be public. Major news and/or revision should be announced and explained. Commentaries in stat releases should be non-partisan. Priviledged pre-release access should be limited. Whenever given, should be controlled and publicised. Ethical standards for management and staff. Statistical confidentiality Protection of confidential statistical information should be stipulated by law, and necessary instructions and IT tools given to staff. ECB reports at least once a year a report on the application of the principle of statistical confidentiality. Comply or explain!
Coordination and cooperation The ECB shall promote coordination and cooperation among ESCB and European Commission experts and other international organisations to support the sharing of information, know-how and best practices. Memorandum of Understanding b/n Eurostat and BCE. Resources endowment and efficiency Major IT innovations and upgraded tools should be available to the producers of statistics to be applied in all steps of collection, production and dissemination of statistics. Staff should be recruited according to high qualification standard, and proper training and continuing education should be guaranteed.
Cooperation b/n Eurostat & ECB Criteria are encompassed in a wider set of quality principles: 1) Institutional Environment 2) Statistical Process 3) Statistical Output
Quality elements of the Statistical Processes Sound methodology and procedures Methodological soundness refers to compliance with internationally accepted standars, guidelines and good practices (Annex to EC1055/08) Methodologies used at ECB are described in the BoP Book A good statistical process should have features like: - early detection of bad data (inaccurate, inconsistent, implausible) - disclosure on aggregation procedures and compilation methods - adequate revision practices (see output quality elements) Cost effectiveness and non-excessive burdens Cost-effectiveness can be evaluated by means of Impact Assessments and enhanced by data sharing and coordination with other statistical collectors: e.g. ECB & Eurostat 2003 agreement)
The four stages of statistical processes 1) Development How well does proposed sources fit with definitions, scope,, timeliness, classifications and valuation required? MEDSTAT III 2) Collection Statistical confidentiality, sources monitoring,non-excessive burden,reliability of the survey framework, sampling criteria 3) Compilation (and revision) Adherence to classifications and definitions. Adherence to revision methods 4) Dissemination Does it fulfil user needs? Proper interpretation instructions are provided? Are dissemination tools adequate (e.g. downloadable spreadsheets etc.)
Cooperation b/n Eurostat & ECB Criteria are encompassed in a wider set of quality principles: 1) Institutional Environment 2) Statistical Process 3) Statistical Output
Quality elements of the Statistical Output Relevance Accuracy and reliability (stability) Consistency and comparability Timeliness and punctuality Accessibility and clarity
Relevance Identify new user requirements consulting main users Monitor users satisfaction Arrange user needs in order of priority Accuracy and reliability (stability) Checks to detect inaccurate, inconsistent or implausible data Stability indicators (size indicators, directional indicators ) Publish stability indicators and analysis of revisions (quality report and self-assessment) Explaining the outliers (where possible)
An example of revision analysis: Source: ECB BoP and IIP Quality Report 2010. Revisions of the Euro area current account. The chart reports the magnitude of the revision as a percentage of the respective item (items are current account flows). All revisions magnitude are declining in time (negative trend is good sign!), although for income flows high revision magnitudes are still the case.
Examples of stability/reliability indicators: Average of absolute revisions: R Directional reliability indicators: n n N 11 22 Q = + Mean absolute percentage error Mean absolute comparative error 1 n n i= 0 1 n n i= 0 R X i i i Ri P
Consistency and comparability Over time Within dataset (bookkeeping approach) Across datasets Across frequencies Mirror consistency Timeliness and punctuality Timeliness: it refers to lapse of time between dissemination date and reference period Punctuality: it refers to the time lag between scheduled dissemination date and actual release. Release calendar has to be public. Delays should be announced and explained.
Accessibility and clarity Statistics and metadata are presented in clear and understandable form Data are published in an appropriate up-to-date electronic format. Excerpts of most relevant series are released in paper publications A desk for assistance to users is provided
Statistics quality stems from multiple dimensions Continuous improvement: learning by doing and users feedback Challenges to data quality: increasing globalisation and cross border activity Conclusions Quality is a complex issue ex post evaluation vs ex ante evaluation Growing importance of IT technologies, especially in web dissemination Growing complexity of the phenomena to be measured: reporting burden vs accuracy dilemma
Giacomo Oddo Banca d Italia Research and International Relations Area Economic and Financial Statistics Department Balance of Payments Division giacomo.oddo@bancaditalia.it