Practical life challenges in implementing data warehouse Presented by Subramanya Hoysala 17 th June 2011
Agenda Overview My Experience Basics in my own way! Challenges Planning and implementing new data warehouse Modifying existing data warehouse Merger and Integration challenges Questions
My Data warehouse experience Creation of new Oracle data warehouse for Variable Life Financial application (VL app) Federation of Oracle data on to a DB2 Data Mart. Recommendation and Implementation of Data Warehouse and ETL for Cash and Custody Application (C&CA)
Data Mart, Warehouse, ODS etc Data Mart: Consists of data that is used by single or few Business entities within the organization Data Warehouse: Consists of the interrelated data that may belong to multiple Business entities within the organization Operation Data Stores: Consists of the interrelated data but for a particular operation Operational Database: Existing databases tuned for faster insert/delete operations
How do you start? Functional Requirements What data user needs Non Functional Requirements How data will be used (Performance, Security, Maintenance, future use etc) Participants needed Subject Matter experts, DBA, Data Architects, and PM
Creation of Oracle DW for VL App (New Data Warehouse Creation) Long process Determine data elements used in existing application Tabulate interrelation and enterprise meaning of data element Metadata creation Model database and apply non functional requirements Finalize database Plan for phased deployment Challenges Focus Large Scope, Broad Vision document Keeping the Momentum Lack of Management push, Other priorities, dwindling funds
Federation of Oracle data (Modifying Existing DW) Application dependent strategy Analyze data needed for Claims system Recommend data model for faster Claims processing Implement real-time update from Oracle to DB2 Challenges Consideration of data inconsistency between two databases Lack of understanding of data usage from other applications
DW & ETL for Cash & Custody App (Merger and Integration Creation of New DW & Modifying Existing DW) Recommendation Analyze the existing databases, processes and applications Recommend the solution for combined entities Compare the future state with current Work with Stakeholders to push for the solution POC for the tool selected Challenges Non functional requirements (Initial load, performance etc) Different priorities (Lack of SME), changes in historic data requirements
Specific to Merger & Integration Understand the dynamics need to analyze multiple options (not only the best options) Collaborate Work with individual units separately and push them towards single solution Evaluate existing tools Learn existing applications and tools Management Support understand the management needs and recommend solutions
Appendix (DW & ETL for Cash and Custody)
Appendix - ETL Process facts (DW & ETL for Cash and Custody) Loading New warehouse is architected using real-time data integration technology Informatica Real Time (RT). Informatica CDC solution and Informatica Data Integration (DI) solutions Informatica Power Exchange implemented on multiple DB2 databases, coupling tightly with DB2 logs. The Informatica CDC components track the changes operational data warehouse for cash, custody, FX rates and pricing, Corporate Action, Tax Reclaims. Any change in data is passed to Informatica PowerCenter RT components. The PowerCenter performs business v alidations on the data and loads to DW Batch Informatica processes are scheduled using ESP scheduler
Appendix (DW & ETL for Cash and Custody) Non Functional Requirements SLA of 30 seconds to report on 10 accounts SLA of 12 Seconds of data consolidation, transformation and migration from source to target for real time reporting and presentation Loading historical data volume (about 320 million records)
Contact Details Subramanya Hoysala Email: shoysala@gmail.com Phone: +1-412-512-7321