Effective Data Integration & Industry Data Model. - Critical Success Factor for Business Intelligence / Data Warehouses IBM Corporation

Transcription

1 Effective Data Integration & Industry Data Model - Critical Success Factor for Business Intelligence / Data Warehouses IBM Corporation

2 Information On Demand is Delivering Value Trusted Information Enables Business Success Mergers & Acquisitions Competitive Pressures Regulatory Compliance Business Efficiency Innovation & Growth Market Dynamics Customer Centricity Enterprise Intelligence Product/Service Optimization Risk & Compliance Business Initiatives Legacy System Consolidations Data Warehousing CDI/MDM ERP/CRM Deployments Major IT Projects Quality Governance IBM Information Platform Mastering Your Data is the Foundation for Success IBM Information Platform Industry Models & Accelerators Industry Models & Accelerators IBM Master Data Server IBM Master Data Server IBM Information Server IBM Information Server Integration Consolidation 2

3 Global 2000 Profiting from Intelligent Information 3

4 Asia Pacific Client List (Partial) Logistics 4

5 Analyst Validation: Gartner s Magic Quadrant "IBM demonstrates the best vision in the market for extensive data integration capabilities, as it continues to progress toward bringing together all its data integration components atop common metadata, common design tooling, and a common look and feel." 5

6 Customers Achieve Significant Productivity Benefits 1 with Effective Data Integration Approx. Project Effort 30% 20% 20% 15% 10% 100% Source System Analysis Data Cleansing Transformation Logic Construction Data Management Services Application System Connectivity 50+% gain 50+% gain 40+% gain 20+% gain 30+% gain 6 1 Compared to hand coding gathered from IBM project studies

7 Costs of Inefficient Data Integration and Data Quality Management 83% of business-intelligence and CRM data integration projects either overrun or fail Scrap and rework Increased costs Lack of consumer confidence Inaccurate or incomplete data is a leading cause of failure 25% of time is spent clarifying bad data Lost opportunities Low data quality costs companies $611 billion annually Undetected defects will cost 10 to 100 times as much to fix upstream 7

8 The IBM Solution: IBM Information Server Delivering information you can trust IBM Information Server Understand Cleanse Transform Deliver Unified Deployment Understand Cleanse Transform Deliver Discover, model, and govern information structure and content Discover, model, and govern information structure and content Parallel Processing Services Standardize, merge, and correct information Combine and restructure information for new uses Standardize, merge, Combine and and correct information restructure information Platform Services for new uses Connectivity Services Metadata Services Unified Metadata Management Administration Services Replicate, virtualize and move information for in-line delivery Replicate, virtualize and move information for in-line delivery Deployment Services Parallel Processing Rich Connectivity to Applications, Data, and Content 8

9 Data Transformation & Movement: WebSphere DataStage Provides codeless visual design of data flows with hundreds of built-in transformation functions Optimized reuse of integration objects Supports batch & real-time operations Produces reusable components that can be shared across projects Complete ETL functionality with metadata-driven productivity Supports team-based development and collaboration Provides integration from across the broadest range of sources Transform Deliver Developers Architects WebSphere DataStage Transform and aggregate any volume of information in batch or real time through visually designed logic Hundreds of Built-in Transformation Functions 9

10 Job Execution Job sequencer for sequencing and controlling job flow DataStage Director 10 Used to validate, schedule, run, and monitor DataStage jobs* Command line interface dsjob run [ mode [ NORMAL RESET VALIDATE ] ] [ param name=value ] [ warn n ] [ rows n ] [ wait ] [ stop ] [ jobstatus] [ userstatus] [ local]

11 Job Monitoring & Logging Detail job monitoring information available during and after job execution Start and elapsed times Record counts per link % CPU used by each process Data skew across partitions Available in the Director Also available from command line dsjob report <project> <job> [<type>] type = BASIC, DETAIL, XML Monitor information at partition level 11

12 Parallel Runtime Execution 12

13 Scalable Performance Benchmark: Scalable Data Integration Using Ascential DataStage Enterprise Edition Rec./Sec. 100,000 75,000 50,000 25,000 1:1 Ratio Linear CPU/Node Note: Contact Ascential for an audited Performance Benchmark Report. 13

14 Change Data Capture and Replication Provides real time changed-data capture and delivery for Dynamic warehousing, ebusiness Synchronization Replication Provides high-volume, low-latency replication for Business continuity Workload distribution Business integration scenarios Minimal impact on production systems High scalability and end-to-end performance Wide breadth of RDBMS support Deliver Developers Transformation Server Replication Server Data Event Publisher ireflect Architects Minimizes impact on performance of production systems 14

15 Technical Metadata: WebSphere Information Analyzer Provides in-depth analysis of existing systems Data-centric analysis of application, database, and file-based sources for content, quality, and structure Secure, detailed profiling of fields, and relationship analysis across fields and across sources Enables ongoing measurement and baseline reporting of information quality Understand Other Product Modules Subject Matter Experts Business Data Analysts WebSphere Information Analyzer Analyze source data Glossary structures, and monitor adherence to integration and quality rules Creates metadata that describes where information is managed across systems Provides an understanding of the fitness of specific sources and highlights data that may need downstream attention Physical View 15

16 Introducing Data Rules for Information Analyzer Adding monitoring to assure accuracy and increase trust Establish Benchmarks for Variance Tracking Create Metrics across single or multiple Data Rules Organize Metrics and Rules within user-defined categories View Metric & Benchmark summaries 16

17 Business Glossary Create and manage business vocabulary and relationships Features Facilitate business & IT communications by creating & managing a common business vocabulary Web based interface shared across enterprise business teams Allows creation of stewards & assignment of their responsibilities for terms & assets. Link business terms / concepts to Electronically Stored Information (technical assets) Benefits Aligns the efforts of IT with the goals of the business Provides business context to information technology assets Establishes responsibility and accountability in accordance with data governance policies Web Browser Subject Matter Experts Analyst Steward Console 17

18 Business Glossary Browser Designed for simplicity read-only access to Business Glossary Features Designed based on two key principles: simplicity lasts and cut right to the chase Read-only browser interface Search and browse the enterprise glossary graphically or textually View details for terms, categories, stewards and other objects Send feedback directly to stewards Benefits Facilitate business-it alignment by encouraging the acceptance and growth of a corporate business glossary Adherence to data governance standards Promotes trust in business glossary assets through collaboration Business Users Simple Search Graphical navigation Web Browser 18

19 Business Glossary Anywhere Real-time access to Business Glossary from any desktop application Features From any desktop application, click on a term & view its business definition in a pop-up window without any loss of context or focus Intelligent matching returns best candidates in a single search Search engine for terms and categories Access steward contact information directly Benefits Increased trust and acceptance of information by delivering definitions in context Expanded adoption of enterprise glossary outside of Information Platform technologies Improved information availability with multiple access mechanisms for electronically stored information (ESI) Pop the Definition! ANY User From Any Application... 19

20 Logical Metadata: Rational Data Architect Data modeling for data structures and federations Federated data discovery Metadata relationship discovery & mapping Subject Matter Experts Architects Rational Data Architect Create and manage business vocabulary and relationships, while linking to physical sources Impact analysis, and synchronization across models SQL & XML generation capabilities Data Modeling & Mapping 20

21 Introducing Information Server FastTrack To reduce Costs of Integration Projects through Automation Business analysts and IT collaborate in context to create project specification Leverages source analysis, target models, and metadata to facilitate mapping process Auto-generation of data transformation jobs & reports Specification Flexible Reporting Auto-generates DataStage jobs 21

22 FastTrack Interface Customizable spread sheet view hosted in metadata repository Source column info Target column info Transformation rule and/or function Drag&drop metadata browser Details of source-to-target mapping DataStage job generation 22

23 Role-Based Tools with Integrated Metadata Business Users Subject Matter Experts Architects Data Analysts Developers DBAs Unified Metadata Management Simplify Integration Increase trust and confidence in information Design Operational Facilitate change management & reuse Increase compliance to standards 23

24 Metadata lineage: IBM Metadata Workbench Web-based exploration of Information Assets generated and used by Information Server applications Cross-tool reporting on data movement, data lineage, business meaning, impact of changes and dependencies Cross-tool tracing of data lineage for Business Intelligence Reports to provide basis for compliance with legislation such as Sarbanes- Oxley and Basel II Understand Data Integration Managers Developers IBM Metadata Workbench Provides IT professionals with a tool for exploring and understanding the assets generated and used by the Information Server suite. 24

25 Where does a field of data in this report come from? Import & Browse Full BI Report Metadata Navigate through report attributes Visually navigate through data lineage across tools Increases trust and understanding of business information Source Tables IBM Information Server 25

26 What happens if I change this column? Show complete change impact in graphical or list form Includes impact on reports in BI tools Allows impact analysis on any object type Reduces the cost associated with IT changes 26

27 Why Should I Care About Cleansing Information? Lack of information standards Different formats & structures across different systems Data surprises in individual fields Data misplaced in the database Information buried in free-form fields Data myopia Lack of consistent identifiers inhibit a single view Kate A. Roberts 416 Columbus Ave #2, Boston, Mass Catherine Roberts Four sixteen Columbus APT2, Boston, MA Mrs. K. Roberts 416 Columbus Suite #2, Suffolk County Name Tax ID Telephone J Smith DBA Lime Cons Williams & Co. C/O Bill st Natl Provident HP 15 State St Orlando WING ASSY DRILL 4 HOLE USE 5J868A HEXBOLT 1/4 INCH WING ASSEMBY, USE 5J868-A HEX BOLT.25 - DRILL FOUR HOLES USE 4 5J868A BOLTS (HEX.25) - DRILL HOLES FOR EA ON WING ASSEM RUDER, TAP 6 WHOLES, SECURE W/KL2301 RIVETS (10 CM) RS232 Cable 6' M-F CandS CS ft. Cable Male-F, RS232 #87951 C&SUCH6 Male/Female 25 PIN 6 Foot Cable The redundancy nightmare Duplicate records with a lack of standards IBM 187 N.Pk. Str. Salem NH I.B.M. Inc. 187 N.Pk. St. Salem NH Int. Bus. Machines 187 No. Park St Salem NH International Bus. M. 187 Park Ave Salem NH Inter-Nation Consults 15 Main Street Andover MA I.B. Manufacturing Park Blvd. Bostno MA

28 Data Cleansing: WebSphere QualityStage Provides specialized data quality processing Ensures clean, standardized, deduplicated information Enables a single version of the truth Supports global postal verification Provides visual tools for designing quality rules and matching logic Seamlessly integrated with DataStage (one engine, one metamodel, one UI) Precisely calibrates matching rules Allows quality logic to be deployed seamlessly within ETL, or as shared services Cleanse Subject Matter Experts Data Analysts WebSphere QualityStage Standardize and correct source data fields, and match records together across sources to create a single view Visual Match Rule Design 28

29 QualityStage Methodology Data Quality Assessment (DQA) Data Re-Engineering (DRE) Investigation Standardization Matching Survivorship Blk 1, 1 St, Frist St, Block 1 1 First Str, # , St, #05-00 Blk 1 First St Blk 1 First St First St # St #05-00 Blk 1 First St Blk 1 First St First St # St #05-00 #05-00, Blk 1, First St #05-00, 1, St 29

30 Illustration of Information as a Service Enters Data / Web Entry Validated and Cleansed Result Calls Data Cleansing/ Scrubbing Web Services Invokes RTI-web service Web Services DataStage + QualityStage for data validation / cleansing / etc 30

31 Logical Metadata: IBM Industry Data Models Industry-proven models, including KPIs and compliance metrics Trusted, single analytical view of the business Proven data model methodology with over 400 clients Accelerated, business-centric development Subject Matter Experts IBM Industry Models Architects Delivers proven industry expertise, models and methodology for six industries with Information Server Models automatically populate and generate metadata in IBM Information Server Reduces project complexity and risk Data Modeling & Mapping 31

32 Information Server and IBM Industry Data Models Banking (Banking Data Warehouse) Financial Markets (Financial Markets Data Warehouse) Health Plan (Health Plan Data Warehouse) Profitability Relationship Marketing Risk Management Asset and Liability Mgmt Compliance Risk Management Asset and Liability Mgmt Compliance Claims Medical Management Provider and Network Sales, Marketing and Membership Financials Insurance (Insurance Information Warehouse) Retail (Retail Data Warehouse) Telco (Telecommunications Data Warehouse) Customer centricity Claims Intermediary Performance Compliance Risk Management Customer centricity Merchandising Management Store Operations & Product Mgmt Supply Chain Management Compliance Churn Management Relationship Mgmt & Segmentation Sales and Marketing Service Quality & Product Lifecycle Usage Profile 32

33 Data Model Impact on Data Warehouse Projects Anticipated Insurance Client Benefits Mitigated project project risk % Fit Fit with with our our business requirements % Savings from from initial initial analysis phase...75% Design Design phase phase (inc. (inc. logical logical and and physical data data models)....65% Biggest Impact on DW Project ROI POC Metrics from Major Electronics Retailer ETL ETL activities..20% 2 to to 4 weeks weeks for for KPI KPI selection to....2 hours hours Anticipated 6 to to reuse 12 reuse 12 weeks weeks savings for for on Logical on Logical next model next model build build to days days project 50% 0% of of useful useful current current KPIs KPIs to %, 1 BST BST Added Added days days for for business metadata capture capture to Minutes 33

34 Plugging Industry Data Models into IBM Information Server Five Ways the IBM Data Model Accelerate DW Development Identify business analysis areas and data requirements Define enterprisewide data definitions and data standards IBM Information Server Unified Deployment Create target data warehouse and mart structures for trusted data Understand Cleanse Transform Deliver 34 Discover, model, and govern information structure and content Explain to business users the definition of the data they re using 34 Standardize, merge, and correct information Parallel Processing Combine and restructure information for new uses Unified Metadata Management Rich Connectivity to Applications, Data, and Content Synchronize, virtualize and move information for in-line delivery Simplify data warehouse design, reuse and lifecycle management Information Server is the ONLY data integration platform capable of exploiting the full value of the data models

35 The IBM Information Server Advantage Simplifying Information Integration IBM Information Server accelerates information integration speed and flexibility by providing: An easily deployable, unified foundation for enterprise information architectures Metadata-driven automation, accelerating productivity and flexibility for integrating, enriching and understanding information Simplified scalability at lower cost to manage current and future data requirements Data governance capabilities to ensure consistent and accurate compliance with information-centric regulations and requirements Broadest and deepest connectivity and platform support to leverage and extend existing IT investments Integrated with IBM Industry Data Model 35

36 Information Platform & Solutions: Fast Track Your Master Data 01.ibm.com/software/data/ips/ Ivan Lee

37 Next step InfoSphere Warehouse Proof of Technology Workshop Sept 18, 2008 (Thur) 2:30 5PM IBM Solution Centre 10/F, PCCW Tower, Taikoo Place, 979 King s Road, Quarry Bay, Hong Kong Demonstrating Cognos, Information Server & InfoSphere Warehouse 37

38 38