A JOURNEY TO TRUSTED DATA

Transcription

1 TELSTRA TEMPLATE 4X3 BLUE BETA TELPPTV4 A JOURNEY TO TRUSTED DATA RICK ANDREWS MAY 2014

2 ENTERPRISE DATA WAREHOUSE THE INFORMATION FACTORY 1 BUSINESS EVENTS occur & raw data is created 2 3 Subassemblies of INFORMATION may be combined PROCESSES to extract, transform and load data are applied 4 FINISHED INFORMATION PRODUCTS delivered to enable decision making All customer data is collected, used, disclosed and secured in accordance with applicable law CORPORATE DASHBOARD SELF SERVE ANALYTICS

3 PERCEPTIONS OF DATA QUALITY

4 DATA DIGITAL LIFEBLOOD OF AN ORGANISATION if data is not MANAGED then it can become a RISKY LIABILITY rather than a VALUABLE ASSET

5 AN OLD MANAGEMENT ADAGE THAT STILL APPLIES TO DATA You can t MANAGE what you don t MEASURE

6 THE SOLUTION A DATA QUALITY FIREWALL PURPOSE To automatically and routinely monitor data in the Enterprise Data Warehouse and to raise alerts when data quality violations are detected ANTICIPATED RESULT Improve the COMMUNICATION OF DATA HEALTH to all EDW stakeholders Business view Technical view Available What Complete When Error-free Scale

7 DATA QUALITY FIREWALL FRAMEWORK OVERVIEW Data Sources Enterprise Data Warehouse BI Platform Source System Source System Source System E T L ESA Files E T L Atomic Data Store E T L Data Marts Data Marts D A T A D E L I V E R Y BI Tools Reports / Dashboards Info Delivery DATA QUALITY FIREWALL Testing Metadata Management Data Standards & Technical Integrity Data Governance & Data Stewardship Rick Andrews Data Quality Firewall Telstra Unrestricted Final

8 HOW IT WORKS THE SIMPLE VERSION Schedule Outputs Items To Be Tested DQ Rules Engine EDW

9 DIMENSIONS OF DATA QUALITY THE CRITICAL FEW AVAILABLE COMPLETE ERROR-FREE CURRENCY TIMELINESS COMPLETENESS INTEGRITY UNIQUENESS VALIDITY Time elapsed since real world entity was recorded or business event occurred Data is sufficiently up-to-date for the task at hand Data is not missing and is of sufficient breadth and depth for its specified use Correct references to related entities (Referential Integrity, Entity Integrity) Entities exist only once within a data set Conformance to business rules for the entity

10 COMPARING ACTUAL TO EXPECTED Simple Medium Complex Single Table Analysis Multi Table Analysis BR Multi Table Analysis

11 BUILD APPROACH Tier 0 Source to Staging Tier 1 Staging Tier 2 ETL Tier 3 ADS Tier 4 Access Layer Tier 5 BI Platform Completeness Consistency Currency Custom Integrity Late Landing Completeness Timeliness Validity Volume Targeted Builds Selected small scale builds based on user requirements Mass Builds Systematic large scale builds based on technical authoritative points of truth

12 AN EXAMPLE OLD VOLUME FORECASTING APPROACH

13 AN EXAMPLE OLD VOLUME FORECASTING APPROACH Wide thresholds masking issues

14 AN EXAMPLE NEW VOLUME FORECASTING APPROACH 85% Confidence Limits

15 DRIVE IMPROVEMENTS IN EDW PHYSICAL DATA MODEL EDW PDM Data Quality Firewall EDW Analysis of violations

16 PROVIDE DEVELOPMENT TEAMS WITH INSIGHT INTO DATA QUALITY ISSUES Initiate Discovery Evolve Operate Data Quality Firewall Rick Andrews Data Quality Firewall Telstra Unrestricted Final

17 TOP 8 OUTCOMES Detecting and alerting on data quality issues Provide development teams with insight into data quality issues Supporting data quality improvement activities Providing evidence that data quality issues have been resolved Drive improvements in EDW Physical Data Model Assisting in prioritisation of remediation activities Identified improvement opportunities in operational processes Drive compliance to standards

18 THE DATA QUALITY FIREWALL HAS enabled us to develop a WINDOW INTO THE QUALITY OF DATA within our Enterprise Data Warehouse This improved visibility enables us to affect user s TRUST in the data and better MANAGE our data

19