Data Integration for Data Warehousing and Data Migrations. Philip Russom Senior Manager, TDWI Research March 29, 2010

Transcription

1 Data Integration for Data Warehousing and Data Migrations Philip Russom Senior Manager, TDWI Research March 29, 2010

2 Sponsor: 2

3 Speakers: Philip Russom Senior Manager, TDWI Research Philip On Director, Enterprise Information Management, SAP 3

4 Agenda Overview Defining Data Integration (DI) The Two Main Practices of DI Repurposing Data, Adding Value Success Factors for DI whether for Warehousing or Migrations Collaboration Common & Unique Skills Organization, Staffing, Management Tools and their Use Recommendations 4

5 Overview Data integration (DI) is an autonomous practice. It s not merely a subset of data warehousing (DW) or database administration (DBA) Though one practice, DI has analytic and operational uses. Analytic DI (AnDI) is mostly applied to data warehousing. Operational DI (OpDI) is mostly applied to data migrations. There are challenges due to DI s autonomy & An/Op split: What kinds of DI projects are typical of this autonomous practice? Should you staff DI separately from DW or DBA? Combine AnDI & OpDI? How does staffing relate to DI sponsorship, funding, training, etc.? How do DI s collaboration requirements vary across practices & projects? What are the skills of broad DI work? Can one person know it all? Can all DI work share common tools and integration platforms? Today s Webinar covers these and similar challenges. Given the growing breadth of DI work, DI specialists and people who depend on them need to think more about how to organize, staff, train, tool, and coordinate DI work. 5

6 Data Integration s Two Broad Practice Areas The two broad practice areas of data integration are distinguished by project types with which they're associated: Analytic data integration (AnDI) is applied most often to data warehousing and business intelligence, less often to similar initiatives, like customer data integration or master data management. Operational data integration (OpDI) most often manifests itself as projects commonly described as the migration, consolidation, collocation, or upgrade of operational databases. It may also involve synchronizing data across operational databases or business-to-business data exchange. Enterprise Data Integration Analytic Data Integration (AnDI) Initiatives commonly supported: Data Warehousing (DW) Business Intelligence (BI) Hybrid DI: Master Data Management (MDM) Customer Data Integration (CDI) Product Data Integration Operational Data Integration (OpDI) Implementations commonly supported: Database Migration Database Consolidation Database Collocation Database Mgt System Upgrade Database Synchronization B2B Data Exchange 6

7 DI Work is Growing Over time, OpDI has grown as a percentage of overall DI work, relative to AnDI: 81% AnDI and 19% OpDI in % AnDI and 49% OpDI in 2008 Rounded: 80/20 to 50/50 split OpDI growth is a dramatic change that only took four years. OpDI needs more people and budget to keep pace with growth. Both AnDI and OpDI are growing. OpDI is growing faster than AnDI. Both practices need resources. DI work is getting more varied and projects are more numerous. DI projects for both BI/DW & op.apps are getting bigger. With data integration usage in your organization, what is the approximate percentage split between analytic DI versus operational DI? November 2004 (Source: Forrester Wave Report) February 2006 (Source: TDWI Tech Survey) February 2007 (Source: TDWI Tech Survey) November 2007 (Source: TDWI Tech Survey) August 2008 (Source: TDWI Tech Survey) December 2008 (Source: TDWI Report Survey) Analy tic Data Integration (AnDI) 61% 63% 51% 81% 75% 75% Operational Data Integration (OpDI) 39% 37% 49% 19% 25% 25% 7

8 Data Integration is the Repurposing of Data via Data Transformations Analytic Data Integration ETL for data warehousing repurposes operational data into data schema appropriate to reporting and analytic applications Data federation does the same, but typically with less complex transformations of the data Operational Data Integration Most data migrations transform legacy data into more modern schema so it s better suited to the purposes of the target platform Data sync and replication typically do light transformations B2B data exchange relies on standards to minimize transformations All DI work is defined by the transformation of data Note data transformation is a technology, and it enables a business goal, namely the repurposing of data. 8

9 Data Integration is a Value-Adding Process This is especially true of Analytic Data Integration. DI is similar to the value-add processes of manufacturing. Diverse raw material is processed & combined to form a new product. The calculated values, aggregates, and dimensional models of a data warehouse are high-value data that doesn t exist elsewhere. Not even in the source IT systems that provided raw material for DI! Operational DI adds value to migrated, exchanged & sync d data. Migration is never mere copying. Data is transformed as it s moved, and sometimes totally remodeled for improvement. Data sync enriches views of customers, products, etc. DI often taps other technologies for data s improvement. DI tools regularly call data quality tools. A good DI solution improves metadata and master data. 9

10 Data Integration is Inherently Collaborative, Regardless of Project Type Analytic DI is a Paradigm of Collaboration Tradition of aligning business intelligence (BI) and data warehousing (DW) to the business: Biz requirements are actively pursued and executed. The data warehouse models the business. The reports and analyses of BI express the state of business activities, as seen from its data. The complex BI/DW/DI technology stack requires collaboration: Many technical disciplines to coordinate: BI, DW, DI, analytics, reporting, data quality, MDM, metadata, modeling, profiling, etc. Design must be collaborative to keep the complex tech stack faithful to biz needs Many of DI s collaborative habits come from BI/DW, others from data quality & stewardship More collaborative possibilities are coming from data governance Operational DI is also Collaborative, though in a different way compared to Analytic DI Database migrations and consolidations Migrations take away old IT systems and introduce new ones. DI collaborates with business units that use them and business managers who own them. Since it s often app databases being migrated, DI collaborates with application developers Migrations usually entail multiple phases and steps. DI collaborates with business people and application team to design and execute the steps. Business-to-business data exchange Core to collaboration between partnering organizations, especially with data-driven supply chains Data synchronization DI collaborates with the people who use, own, and design the databases/apps being sync d. 10

11 Data Integration Involves Common, Core Skills, Plus Skills that Vary per Practice or Project Type All DI work assumes a deep knowledge of: Relational databases (sometimes legacy databases, too), vendors database management systems, SQL and similar languages, common interfaces both standard and proprietary Operational DI requires skills with applications & DBA work Operational, transaction, and other process-driven applications; replication & synchronization; data exchange standards; unstructured data Analytic DI requires skills with data for reporting & analysis Multi-dimensional databases and data structures (e.g., star & snowflake schema); slowly changing dimensions; modeling, merging, and loading multi-dimensional data 11

12 Data Integration s Success Depends on Proper Staffing, Management, and Organizational Support Organizational support Analytic data integration Usually staffed by BI/DW team, funded by its sponsor Operational data integration Often staffed by a data management group, less often by database specialists on an applications team Staffing Problems When DW teams do OpDI, it delays AnDI work & upsets DW sponsors Staffing separate teams for AnDI & OpDI yields redundant personnel TDWI s Take on Staffing DW and BI are ongoing, requiring permanent Analytic DI staff 74% of Respondents say OpDI work is continuous or predictable So, staff it separate from DW team, perhaps with a DI Competency Center When OpDI work is intermittent Outsource it to consultants, vendors, etc. 12

13 Data Integration Solutions are usually Strategic, Whether the Business Recognizes it or Not Analytic data integration Most organizations consider their data warehouse to be strategic. Likewise, they consider strategic the reports and analyses fed from the data warehouse. Without data integration, there wouldn t be data in the warehouse or in reports and analyses. Therefore, analytic data integration is strategic. Operational data integration Mergers and acquisitions are a key strategy for growth and competitiveness. So are reorganizations. DI migrates/consolidates data for these, so it s strategic, too. Most business-to-business partnerships depend on exchanged data, so OpDI contributes to strategic partnerships. Likewise, 360-degree views of customers, products, etc. support enterprise strategies, and OpDI s data sync enables these. 13

14 DI Tool Issues In general, build DI solutions atop a vendor s tool Hand coding is nonproductive, feature-poor, not modern, not maintainable, lacks interoperability, high payroll cost. Object/project reuse and data/dev standards more likely with tool. For max interoperability, growth, reuse, and standards, acquire as many DI and related tools (DQ, MDM, DG) as possible from a single vendor. IMPORTANT: Many DI licenses are for Single Use If licensed for BI/DW, using tool for OpDI may require another license. AnDI has one set of unique tool requirements: DI tool needs strong support for complex transformations, multi-dimensional data structures, slowly changing dimensions, materialized views, data federation, etc. OpDI has a slightly different set of tool requirements: DI tool should support data exchange standards, access to legacy DBMSs, change data capture, data replication & synchronization, etc. When OpDI work is intermittent, look for tools available via: Short-term licenses, a DI cloud, software as a service (SaaS) 14

15 Summary and Recommendations Recognize Data Integration s autonomy and diversity DI is an autonomous practice, not a subset of DW or DBA work Operational Data Integration are Analytic Data Integration two separate practice areas All DI work has common goals, regardless of the practice areas or project types where it s applied DI repurposes data via data transformations DI is a value-adding process DI is inherently collaborative, though the scope of collaboration varies DI is strategic when it supports strategic biz and IT initiatives Some DI requirements vary, depending on practice areas or project types All DI work assumes database and interface knowledge But AnDI and OpDI each involve a few unique skills AnDI is tied to the BI/DW technology stack; OpDI to operational applications This affects skill sets, required interfaces, people to collaborate with, etc. DI Tools demand special consideration Use a tool, not hand coding, for greater productivity, features, maintenance, etc. Use tools from a single vendor for all DI work, for good reuse & standards Note that OpDI and AnDI projects may require separate tool licenses When DI work is intermittent look for DI tools via short-term license, cloud, SaaS 15

16 One last recommendation Download a free copy of the TDWI report on Operational Data Integration Download the report in a PDF file at: TDWI.org, then select Research, then Best Practices Reports Distribute the report s PDF file freely 16

17 SAP Solution Overview Enterprise Information Management Philip On Director, Enterprise Information Management SAP March 29, 2010

18 Poorly Managed Information Leads to Inefficiency and Risk Over 51% of organizations estimate data related issues cost their company over $5 million. Forbes Insight 90% of all businesses still do not have sufficient policies in place to meet data governance regulations. IT Policy Compliance Group

19 Ensuring Consumer Safety and Increasing Profitability by Being Metrics-driven

20 Build an Information Driven Organization Improve business insight and decision making Increase operational efficiency and reduce costs Meet compliance and regulatory requirements Provide all users with data that is complete, accurate and accessible Provide high quality data to all business processes Enhance information governance via policy-based data management

21 SAP Provides Best-In-Class EIM Solutions Deliver Information That Is Complete, Accurate, and Accessible Data Integration & Quality Management: SAP BusinessObjects Data Services SAP BusinessObjects Data Federator SAP BusinessObjects Text Analysis SAP BusinessObjects Data Insight SAP Data Migration services Master Data Management: SAP NetWeaver Master Data Management SAP Master Data Governance for Financials SAP Data Maintenance by Vistex Content & Information Lifecycle Management: SAP NetWeaver Information Lifecycle Management SAP Extended ECM by Open Text SAP Document Access by Open Text SAP Archiving by Open Text Enterprise Data Warehousing: SAP NetWeaver Business Warehouse SAP NetWeaver Business Warehouse Accelerator SAP BusinessObjects Rapid Marts SAP BusinessObjects Metadata Management SAP 2007 / Page 21

22 Market Leadership Industry Recognition and Customer Success Best rated solution by industry analysts Data Integration Product of Year Award by Searchdatamanagement.com Rated as a market leader in data quality and data integration by Gartner, Forrester, and TDWI Rated number one for ease of use to accelerate time-to-delivery by Passionned Research SAP customer, Peachtree Data, wins innovation award by Information Management SAP customer, Kraft Foods Inc., wins Gartner MDM Excellence Award SAP 2007 / Page 22

23 Improve Business Insight Ensure that Information is Complete and Accurate Integrate to provide access to all information SAP BusinessObjects Data Services integrates and ensures high quality data from any environment SAP BusinessObjects Data Federator enables high performance views into any data source Build an information repository SAP NetWeaver Business Warehouse enables highly scalable enterprise data warehouses SAP NetWeaver BW Accelerator provides access to massive amounts of information at Google-like response times SAP BusinessObjects Rapid Marts jump starts data mart deployments for popular SAP and Oracle applications SAP 2007 / Page 23

24 SAP BusinessObjects Data Services Integrate and Improve the Quality of All Data Market-leading, unified solution for enterprise-class data integration and data quality Single, easy-to-use user interface to build, test and deploy projects Connect, transform and make the information available from virtually any sources Comprehensive data quality solution for cleansing and enriching all types of data Embed data quality steps directly into the ETL process to build a data warehouse Having all this functionality as part of one application makes it easy to select from the different data transforms provided the software and run data through them quickly and efficiently. Richard West President, Peachtree Data Inc. SAP 2007 / Page 24

25 SAP BusinessObjects Data Federator Gain unified and real-time view of disparate information Data Federator BI Universes OLAP access Pull information on demand from multiple data systems and applications into a unified view. Access real-time information from multiple data sources using data federation Get fast information access from disparate operational systems with optimized query performance Non invasive, safe technology low impact on source systems Database SAP NetWeaver BW Wide source support of relational and non-relational systems such as RDBMS, XML, SAS, Web Services, and SAP NetWeaver BW Enable data lineage and analysis from report to data source with Metadata Management SAP 2007 / Page 25

26 Increase Operational Efficiency Provide High Quality Data to Critical Business Processes Measure and improve data quality SAP BusinessObjects Data Quality Management to profile, cleanse and improve data Manage master data across the enterprise SAP NetWeaver Master Data Management to harmonize and centrally manage master data SAP Master Data Governance for Financials to consolidate chart of accounts across the enterprise SAP Data Maintenance for ERP by Vistex to minimize data maintenance of key resource and pricing data Migrate data for SAP ERP implementations SAP Data Migration services provides proven mythologies to reduce risk in migration projects SAP 2007 / Page 26

27 SAP Data Migration for SAP ERP Complete Offering to Accelerate Data Migrations 80% of data migration projects in support of new or upgrade enterprise applications run late or over budget. Bloor Research on Data Migration SAP DATA MIGRATION ANALYSIS EXTRACT CLEAN VALIDATE LOAD RECONCILE Governance and Visualization SAP BusinessObjects technology Proven software and expertise in data migration, data quality and data integration and data warehousing SAP Services Proven functional expertise and data loading into SAP ERP Pre-defined content for configuration extraction. Extract rules from target system into SAP. SAP 2007 / Page 27

28 Why SAP? The Best Choice for EIM Time to Value: Fast and cost-effective integration with existing SAP and non-sap systems Proven Customer Value: Mature offering and large install base of customers supporting critical business scenarios Market Leadership: Analyst recognition and customer implementation success Comprehensive Solutions for EIM Strategy One-stop for end-to-end information governance and management SAP 2007 / Page 28

29 Questions?? 29

30 Contact Information If you have further questions or comments: Philip Russom, TDWI Philip On 30