International Household Survey Network (IHSN) Accelerated Data Program (ADP) Experiences and benefits of implementing the DDI in data management

Transcription

1 International Household Survey Network (IHSN) Accelerated Data Program (ADP) Experiences and benefits of implementing the DDI in data management François Fonteneau, OECD/PARIS21 UNESCAP Expert Group Meeting, Bangkok, June 2011 Rationale DDI benefits DDI in brief DDI users DDI applicability IHSN/ADP and the DDI Contents Processes and IHSN/ADP tools Results obtained Lessons learned Way ahead

2 Rationale Survey data: growing supply, but unsatisfied demand Frequent complains by users: Visibility / we didn t know they had data Accessibility / they don t want to share them Reliability / these data are messy Relevance / this is not really what I need Comparability / how can I assess progress? Usability / where is the documentation? DDI benefits DDI can help : If you want a way to make documenting and sharing your data easier If you are interested in using a common documentation standard that permits Web display If you want to repurpose your documentation to create PDF files, and other products DDI is a way to standardize documentation and make it interoperable. DDI uses XML to create documentation that is structured yet flexible.

3 DDI in brief The Data Documentation Initiative (DDI) is an effort to create an international standard for describing social/economic data. Expressed in XML, the DDI metadata specification supports the entire data life cycle. DDI metadata accompanies and enables: conceptualization, collection, processing, distribution, discovery, analysis, repurposing, archiving.

4 DDI in brief To ensure continued support and ongoing development of the standard, DDI has been branched into two separate development lines: DDI Codebook (formerly DDI2) IHSN/ADP DDI Lifecycle (formerly DDI3) Tools: a variety of tools have been developed or are under development for working with DDI The DDI Alliance ( is a self sustaining membership organization that develops and promotes the DDI specification and associated tools. Member Institutions / Expert Committee / Steering Committee / Working Groups

5 DDI in brief : before and after (...or worse!) DDI users DDI at work: From:

6 DDI: applicability National and international statistical organisations Data producers (NSOs, etc.) and users (researchers, data archives) Microdata ; independent of data source, can be used for: Surveys / censuses Administrative sources / register based statistics Mixed sources IHSN / ADP...and the DDI Second Intl Roundtable on Managing for Development Results (2004, Marrakech): adoption of the Marrakech Action Plan for Statistics (MAPS): creation of IHSN and ADP IHSN: to improve the coordination of intl. organisations household survey activities. ADP: to help data producers in developing countries increase the use and value of their survey data. Program funded by World Bank, administered by PARIS21, implemented by a small core team (P21 and WB); open partnership (UNICEF, DFID, SPC, FAO, WFP, others)

7 IHSN / ADP...and the DDI Better use of existing microdata Data producers are not always the most interested and qualified users. More users more diverse, better use. Microdata preserved and documented in countries according to international good practices All surveys and censuses catalogued Formal dissemination policies and procedures, compliant with national legislations Better data collection Improved quality of future surveys Processes and IHSN/ADP tools Establishing a survey data archive involves : Acquisition [1,4] inventory, t.o.use Preparation (incl SDC) [5,6] various Documentation [all process] MMT Cataloging [7] NADA Dissemination [7] NADA+WP5 Preservation of data and metadata [8] WP3 [GSPBM phases] 6 36 months implementation in countries!

8 Tools demo DDI survey metadata editor Microdata Management Toolkit DDI web based survey cataloguing system National Data Archive application

9

10 Results : survey catalogs / Asia Cambodia National Institute of Statistics Lao PDR Department of Statistics Mongolia National Statistical Office Philippines National Statistics Office Bureau of Agricultural Statistics Bureau of Labor & Employment Statistics Sri Lanka Department of Census & Statistics Viet Nam General Statistics Office Coming Soon: Nepal, India Lessons learned (1) Quality management: good documentation critical Know what you have done to evaluate it and do it better Challenging: iterative process involving different organizational units in statistical agencies Key to improve data use and Users satisfaction

11 Lessons learned (2) DDI a very useful standard; user friendly Huge demand/interest in NSOs (on going in 60+ voluntary countries) Huge demand: sometimes terrible situation : Several weeks to gather documentation for one survey Files lost (high staff turnover, heavy reliance on external TA) Little interest in metadata reuse (survey planning) Many challenges: Fear of change Multiplicity of standards and solutions silo mentality : lack of harmonization in terminology, methodology, tools NSO structures not based on processes Issue : organize / internalize data documentation within agencies Cultural more than technical challenge See UNECE Part A / Common Metadata Framework Statistical metadata in a corporate context : guidance on the issues surrounding the management and governance of statistical metadata system Way ahead Keep promoting documentation activities Promote existing standards Encourage TA / funding partners discipline and alignment Promote the adoption of formal dissemination strategies SDC: practical solutions still require some work

12 More? IHSN: ADP: DDI: