The Data Quality Program in PEDSnet. Ritu Khare, PhD Department of Biomedical and Health Informatics The Children s Hospital of Philadelphia

Transcription

1 The Data Quality Program in PEDSnet Ritu Khare, PhD Department of Biomedical and Health Informatics The Children s Hospital of Philadelphia

2 Contents PEDSnet Overview Data Quality Results (JAMIA) Key Data Quality Checks

3 700 #OMOP table records 600 (in millions) PEDSnet CDRN = 5.86M patients in pediatrics

4 PEDSnet Data Cycle S1 S2 S3 S4 S5 S6 S7 S8 PEDSnet sites (at various locations) PEDSnet ETL Conventions (v2.7) ETL OMOP Common Data Model Data Quality WorkUlow PCORnet ETL Specifications (v3.1) ETL PCORnet Common Data Model

5 Data Quality Conceptual Model Check Type Check Missing data Missing data for person.gesta)onal_age designed for OMOP Domain contains Field identifies Site X observed 70% missing data for gesta)onal age in data cycle Y Data Cycle Nota%on Entity relationship Issue observed at Site generaliza)on

6 PEDSnet Data Quality (DQ) WorkBlow (2) Investigate differences (1)Apply DQ checks Site data (OMOP ) (3) Post feedback Site ETL Analysts GitHub (4) Identify causes DQ Issues (current) DQ Issues (previous) Data Quality warehouse

7 GitHub Issue Screenshot

8 Results Results of Data Quality

9

10 Methods PEDSnet Data Quality Warehouse January 2015 March 2017 (13 data cycles) Total 9,086 data quality issues and related metadata OMOP domain Field Check type Cause Identi2ied (all) vs. reported(new) issues time to closure (GitHub)

11 Checks vs. Issues 1400 OMOP v4 OMOP v Checks All Issues Data Cycle Pilot cycle Code review Code refactoring

12 Causes across issues % of all Issues i2b2 transform Under Review False alarm Characteristic ETL Data Cycle

13 Distribution across Domains (cycle-13) Total number of issues

14 Distribution across Check Types (cycle-13) Total number of issues

15 All issues = Persistent + New (reported) issues 1400 OMOP v4 OMOP v Number of Data Quality Issues New issues Persistent Issues Data Cycle Access to additional units/data, i2b2- >OMOP

16 Causes across new(reported) issues Percentage of New Issues i2b2 transform Under review False alarm Characteristic ETL Data Cycle

17 Time to closure of GitHub issues

18 Results Data Quality Checks

19 Evolution of Data Quality Checks Number of Checks THEORY-DRIVEN Frameworks, methods in literature (Brown et al. 2013, Weiskopf and Weng, 2013, Kahn et al. 2015) c Ø PESDnet committee members Ø Data and issue review DEVELOPER-DRIVEN PEDSnet engages in science (>50 studies) USER-DRIVEN Data Cycle

20 Data Quality Checks in PEDSnet Overlap with Achilles Heel Value set violation Invalid concept identibier Illegal vocabulary Missing data No matching concepts Future event Pre-birth fact Post-death fact Start date after end date

21 Data Quality Checks in PEDSnet New Checks (PEDSnet data committee) Inclusion criteria violation Date time inconsistency measurement_datetime vs. measurement_date Invalid format procedure_source_value, condition_source_value Unexpected change between data cycles number of records missingness in Bields

22 Data Quality Checks in PEDSnet New checks (science queries) Missing expected concept E.g. creatinine labs, nephrology specialty for providers. InsufBicient facts for specibic visit types E.g. missing DRGs for inpatient admissions Unexpected more frequent values Identify outliers in top conditions and procedures (using crosssite comparison)

23 Open Questions and Challenges in Check Design Design checks for new (unexpected) issues encountered during science queries Determine the combination of Bields / tables Determination of thresholds Automatic review of ETL mappings labs, organisms, specialty, route, race, ethnicity, drugs, language, procedure, smoking history 1000s of manually derived mappings PEDSnet Data Quality Checks available on GitHub

24 Acknowledgments Data Quality Team David Soler Josh Tucker PEDSnet data scientists Hanieh Razzaghi Levon Utidjian PEDSnet DCC director L. Charles Bailey Other PEDSnet teams ETL analysts Site Informatics Leads Leadership and Governance PCORnet Governance Committees and DRN OC This work was supported by PCORI Contract CDRN OHDSI Consortium Patients and Families