Data Aging with mysap Business Intelligence

Size: px

Start display at page:

Download "Data Aging with mysap Business Intelligence"

Ilene Hunter
5 years ago
Views:

Data Aging with mysap Business Intelligence Rainer Uhle, Product Management BI, SAP AG Learning Objectives As a result of this lecture, you will be able to: Describe the main challenges of

1 Data Aging with mysap Business Intelligence Rainer Uhle, Product Management BI, SAP AG Learning Objectives As a result of this lecture, you will be able to: Describe the main challenges of Information Lifecycle Management Understand SAP s overall Archiving Strategy for ERP and DW Components Explain the benefits of Near Line Storage in an SAP BW environment Appreciate that TCO can be dramatically decreased for your data warehouse by a successful archiving strategy Understand that SAP Business Intelligence provides the rich toolkit for implementing your archiving and near-line storage strategy. SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 2

2 Agenda Information Lifecycle Management Information Lifecycle Management Data Aging Strategies in a Heterogeneous Application Environment SAP ERP Components SAP Business Information Warehouse Summary SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 3 Agenda Information Lifecycle Management Information Lifecycle Management Data Aging Strategies in a Heterogeneous Application Environment SAP ERP Components SAP Business Information Warehouse Summary SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 4

3 Increasing Volume of Data Bases Meta Group 70% of all enterprise data currently resides in databases Gartner Group Multi-Terabyte databases are a reality today and will grow to hundreds of TB by 2006 Database applications like SAP are a key driver of storage growth Growing at 64% rate SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 5 The Attractiveness of Data is changing with its age Frequency of access reads updates Age of data SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 6

4 Distribution of the memory costs Hard disk costs do not even represent a quarter of the memory costs (Giga Information Group) Misc (Purchasing, training) 10% Personel 45% Hard disk 23% Environment (Electricity, Space) 3% Storage- Mangement (Soft- & Hardware) 19% Administrative expense for 1 Terabyte of memory are appropriate for five to seven times more higher than the memory costs themselves (Dataquest/Gartner) SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 7 Data Lifecycle vs. Storage Media Types/Costs Access Frequency high low Cost time RAM DASD Virtual Tape Access Centric Tape Capacity Centric Tape Performance Online- Storage days months years Near-line Storage Offline Archive SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 8

5 How to combine all this in a Data Aging Scenario? Query Online File System Offline/ Near-line Read Database Data Objects Application Data Archive Files Storage Management System Move Save SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 9 Benefits of a sound Data Archiving Strategy Value proposition for deploying Data Archiving 1Availability Faster and simpler software and 2 release mgmt and upgrades. Reduced backup & recovery Resource consumption times. Reduction of the hardware costs for hard disks, main memory and CPU, and the costs of the system administration 3 Performance Faster Dialog response times. Faster Data load times. SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 10

Information Warehouse Summary SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle /

6 Agenda Information Lifecycle Management Information Lifecycle Management Data Aging Strategies in a Heterogeneous Application Environment SAP ERP Components SAP Business Information Warehouse Summary SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 11 Open Data Warehouse Architecture SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 12

BW Architecture: Layers & accessibility Operational Data Store Operational Reporting Near Real-Time / Volatile Granular Built with

Multidimensional analysis Aggregated view Integrated Built with InfoCubes SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle /

InfoCubes Customer dimension A InfoCube Material dimension 4712 4713 Amount Customer Material Time Company Currency Information A

Documents Change Docs Customer Time Doc No Pos Material Amount Local Currency A 20021107-10am 1 10 4711 100 - New booking A

7 BW Architecture: Layers & accessibility Operational Data Store Operational Reporting Near Real-Time / Volatile Granular Built with ODS Objects Data Warehouse Non volatile Granular Historical foundation Integrated Built with ODS Objects Multidimensional Model Multidimensional analysis Aggregated view Integrated Built with InfoCubes SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 13 Transaction Data Processing into the BW MM Layer SAP BW InfoCubes ODS Object ODS Objects BW Architected Data Mart Layer other InfoCubes Customer dimension A InfoCube Material dimension Amount Customer Material Time Company Currency Information A A Time dimension other InfoCubes monthly weekly daily SAP ERP daily Process Data Master data Documents Change Docs Customer Time Doc No Pos Material Amount Local Currency A am New booking A pm Correction booking A pm New booking SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 14

8 SAP BW Transaction Data Processing with BW EDW Layer InfoCubes BW Architected Data Mart Layer other InfoCubes Customer dimension InfoCube Material dimension A 4712 Customer Material Time Amount Company Currency A A Time dimension Information other InfoCubes monthly weekly BW Enterprise Data Warehouse Customer Time DocNo Pos Material Layer ODS ObjectODS Objects Amount Local Currency daily Amount Company Currency A am A pm A pm Information Base SAP ERP Master data Documents Change Docs daily Process Data Customer Time DocNo Pos Material Amount Local Currency A am New booking A pm Correction booking A pm New booking SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 15 Impact of Information Base EDW Layer Pros Designed for the future Foundation of data for future developments Easier delta load control. Reduce redundancy in very granular data Greater control of data distribution Fine tuning of data availability in InfoProviders Archiving of base data possible Cons Increasing Data Volume Data redundancy Increased data management and coordination Increased DW administration Upfront design considerations SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 16

9 Bill Inmon s: Enterprise Data Warehousing Departmental Data Marts Acctg Marketing Finance Sales SAP BW Powered by SAP NetWeaver DSS Applications ERP ERP ERP ETL CRM Changed Data Staging Area EDW ecomm. Bus. Int. ERP Corporate Applications local ODS Global ODS Oper. Mart Granularity Manager cross media Storage mgr Exploration warehouse/ data mining Session Analysis Near line Storage Internet Dialogue Manager Cookie Cognition Preformatted dialogues Web Logs Archives Source:Bill Inmon SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 17 Agenda Information Lifecycle Management Information Lifecycle Management Data Aging Strategies in a Heterogeneous Application Environment SAP ERP Components SAP Business Information Warehouse Summary SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 18

10 Data Archiving and Storage within ERP Environment mysap System External Storage System Database ArchiveLink Storage System Application data Data Objects File System HSM- System Alternative Storage SAP Third Parties (optional) SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 19 Archive Development Kit ADK Development tool for data archiving solutions From SAP Integration of archiving functionality into SAP applications From customers Archiving solutions for customer tables and enhancements Application data Database mysap Applications ADK Code page adjustments, alphanumeric format, structural changes, compression, file handling, job control, Archive Files SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 20

11 The Archiving object Definition of the data associated to logical work units All programs that are necessary for archiving, e.g. Testing, Reading, Writing and Delete programs Customizing of data selection and processing Archiving Object Customizing Data Program SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 21 Data reduction in the ERP field! Is the data still needed? Yes Can Data be consolidated? Yes Consolidate Data! No No Yes Access to Productive data switched off! No Can the data be archived? Yes Data remains in the database! Archive Data! SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 22

Accessing Archived Data Database SAP System Search DB + Archive Business processes DRB AS Application Display work units Archive files 100% ADK Index / Archive info structure Indexing Search on

12 Accessing Archived Data Database SAP System Search DB + Archive Business processes DRB AS Application Display work units Archive files 100% ADK Index / Archive info structure Indexing Search on archived data Technical display for all Archive object Display of Object lists SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 23 Archive Information System Flexible creation of Archive data indices Freely configurable info. structures are stored in transparent DB tables Manually built with selected archive files Automatically built during the delete phase of an archiving run Search of archived data Technical display for all archiving objects Display of object lists Based on info. structures Ad-hoc display of selected archive files Creation of display variants Generic tools for customer developments SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 24

13 Agenda Information Lifecycle Management Information Lifecycle Management Data Aging Strategies in a Heterogeneous Application Environment SAP ERP Components SAP Business Information Warehouse Summary SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 25 Typical Data Warehouse Problems End-User Challenges Making timely, informed business decisions - Users cannot wait for historical data to be restored - Transparent access to data for regular reporting and ad-hoc analysis IT Management Challenges Meeting end-user data demand while managing cost - High costs of adding/managing online disk storage - High costs of backup and recovery especially when data is infrequently accessed - Data protection and availability SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 26

Motivation for a Data Aging Strategy : Benefits Costs offline vs. online storage costs System usage overhead CPU, Memory, etc Control of system growth System Availability vs.

14 Motivation for a Data Aging Strategy : Benefits Costs offline vs. online storage costs System usage overhead CPU, Memory, etc Control of system growth System Availability vs. costs Data availability faster rollups, change runs, etc System availability less downtime for backups, upgrades, etc Performance vs. costs Faster load times Faster query times See also "Scalability with SAP Business Information Warehouse at Legal Requirements SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 27 SAP BW Customer motivation to archive Customer requirements (derived from ASUG): Data objects relevant for archiving InfoCubes ODS Objects PSA Master data Functionality Both Archiving and Data Deletion (without archiving) Select data based on any criteria Automatically scheduled on a periodic basis Restoring of archived data Data retention time 3 to 5 years in InfoCube and ODS Objects Consistent archiving processes SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 28

15 How to avoid high Data Volumes in a DW Environment? Is Data still needed? Yes Can the Data be aggregated? Yes Aggregate Data! Maybe No Yes Can the data be archived? No Yes We need to address the no longer needed AND the maybe scenario! Data remains in the database! Archive Data! SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 29 Data Aging Strategy Implementation Data aging is a strategy for managing data over time, balancing data access requirements with TCO. Each data aging Strategy is uniquely determined by the customer s data and the business value of accessing the data. Which tools should I consider to use? Online Database Storage Near line Storage (BW) - Data Archiving Frequently read /updated data Infrequently read data Very rarely read data SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 30

16 Where is Archiving and Near-line Storage applicable? Access frequency reads Decreasing TCO Archiving with SAP BW Archiving (BW 3.X) For analyses, archived data must be reloaded first again into the BW database Reduction in costs of data retention on alternative media Access frequency reads Decreasing TCO Data age Near Line Storage Near-line Storage (BW 4.X*) Direct accesses to data to alternative storage media for Queries Performance and data retention costs for access aged data can be minimized Data age * Pilot project possible for BW 3.X SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 31 SAP BW Archiving Object Architecture activates BW Repository generates Archiving Object InfoCube ODS Object DataManager reads deletes Write Delete Schedules Read Archive Administration (SARA) Datamart Extractor ADK File system, CMS, HSM SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 32

17 Archiving in BW versus other SAP Components SAP BW Technology platform for (analytic) applications Generated data structures Generation of Archiving Objects Mainly replicated data Highly consolidated data Flat archive data objects Selective one-step deletion after complete verification of archived data No special check for archivability Near-line Storage Other SAP components Dedicated to a single Application Delivered (fixed) data structures Delivered Archiving Objects Often original data Unconsolidated (local) data Segmented data objects Individual deletion of verified data objects Check of archivability for individual data objects prior to the archiving run Archive Information System and Document Relationship Browser SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 33 SAP BW data archiving technology : Write process Time Slot archiving is a powerful tool for selection of time dependent transactional archive data. ODS Object Sep 2002 Oct 2002 Nov 2002 Dec 2002 Jan 2003 Feb 2003 Mar 2003 Current Date: Mar 2003 Time slot archiving: Archive complete Years 2002 & 2003 Only complete fiscal years Exclude Nov 2002 Protect archive areas Archive file Sep 2002 Oct 2002 Dec 2002 Pros: Complex time selection options Cons: Limited to time selections only New Data loads Dec 2002 Mar 2003 SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 34

18 SAP BW data archiving technology : Write process Field Data selection archiving is a very powerful tool for free selection of transactional archive data based on any criteria: ODS Object comp 10 Oct 2002 comp 10 Nov 2002 comp 20 Nov 2002 comp 10 Dec 2002 comp 20 Jan 2003 comp 20 Feb 2003 comp 20 Mar 2003 Current Date: Mar 2003 Data selection archiving: Archive Years 2002 & 2003 Company 10 only Archive file comp 10 Oct 2002 comp 10 Nov 2002 comp 10 Dec 2002 Pros: Flexible data selection Cons:Time selections more complex No protected archiving areas New Data loads Comp 10 Dec 2002 Comp 20 Mar 2003 SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 35 SAP BW data archiving technology: Extraction/Reloading BW Extractors and Export DataSources are Archive enabled InfoCube ODS object Update Rules InfoPackage Export DataSource InfoCube Archive Files ODS object InfoPackage is extended by an option 'archive selection' Selection options for available archive sessions and files Only full extraction supported Archive files are scanned with selection criteria of the request Reload to original DataTarget is possible but not recommended Reload Recommendation: Extract to a copy of original DataTarget instead Use MultiProvider to combine remaining data with reloaded data SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 36

19 Why would we consider Near-line storage? Reasons for Direct Query Access to Archive Data Business-driven reasons Introducing new characteristics with a historical background Strategic Analysis of data over long periods Just because it s our data! Legal reasons Regulatory and industry specific requirements Data is immediate accessible from a legal and technical perspective Example: GDPdU demands a ten-year period for retention for tax relevant data and the data s immediate legibility and machine evaluation within the entire period for retention ( 146 u. 147 AO) Example: Food & Drug Administration (FDA) SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 37 Near-line Storage Solution for SAP BW (Key points) Separating frequently used (kept in database) and infrequently used data (stored in Near-line Storage) Supporting both InfoCubes and ODS Objects Transparent access to "non-archived" and "archived" data for queries Hierarchical Storage Management (depending on the provider) First Level: BW Database Second Level + further Levels: Near-line Storage Intelligent Data Access Data Selection Analysis and Feedback High level index in BW DB Low level index in Near-line Storage Openness StorHouse / FileTek, CBW / PBS Software, DiskXtender for BW / Legato, SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 38

Party Providers Virtual InfoCube Near-line Storage Adapter Near-line Storage Partner Solution BW Database High speed disk Robotic

20 Overview of SAP BW 3.X Project Solution Query No Intelligence Basis Cube or ODS Object Query MultiProvider Copy Proprietary Interfaces to select 3 rd Party Providers Virtual InfoCube Near-line Storage Adapter Near-line Storage Partner Solution BW Database High speed disk Robotic Tape Libraries NAS or Low-Cost Disk Optical Jukeboxes SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 39 SAP BW 3.X Project Solution Sample screen shot 3 rd Party Options SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 40

21 SAP BW 4.x Open Solution (Planned) Query Data Flow Control Flow InfoCube/ODS Object with Near-line Services Split and Dispatch Union High level index BW DB Interface Archive/ Restore Near-line Storage Adapter BW Data Base Near-line Storage Partner Solution High speed disk Low level index Data Manager Robotic Tape Libraries NAS or Low-Cost Disk Optical Jukeboxes SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 41 SAP BW Near-line Storage technology A data view of BW and Near-line Storage solution integration ODS Object Sep 2002 Oct 2002 Nov 2002 Dec 2002 Jan 2003 Feb 2003 Mar 2003 Relocate complete Year 2002 Virtual InfoCube FTP Near-line DB table Sep 2002 Oct 2002 Nov 2002 Dec 2002 Dec 2002 Multiprovider - Provides consistent view of data New Data loads Mar 2003 BW Queries SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 42

22 Agenda Information Lifecycle Management Information Lifecycle Management Data Aging Strategies in a Heterogeneous Application Environment SAP ERP Components SAP Business Information Warehouse Summary SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 43 What we plan to deliver in the future Non-SAP Source Operational Data Store SAP Source Persistent Staging Area Data Warehouse Architected Data Marts Information Access New! Archiving & Near-Line Storage SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 44

23 What about your Data Volume growth? Expected size without Archiving Allocated DB size Allocated DB content 'Without' Archiving Initial Archiving With regular archiving DB growth: ~15 GB/month Reduction: ~60GB DB growth: ~7 GB/month SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 45 The right time to start... 1The healthy System Data Archiving shouldn t be the final step to prevent a system going into cardiac arrest! 2Early Planning Proactively maintaining and sustaining performance in the system 3Interdisciplinary Process Data Archiving makes a high level of coordination necessary between IT (technical) and Application (functional) groups SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 46

Further Information Public Web: www.sap.com Services Rapid ROI and Low TCO SAP Customer Services Network: www.sap.com/services/ Consulting Contact Roy Wood, VP SAP NetWeaver Consulting Practice (R.

24 Further Information Public Web: Services Rapid ROI and Low TCO SAP Customer Services Network: Consulting Contact Roy Wood, VP SAP NetWeaver Consulting Practice SAP BW 3.x Near-line Pilots contacts Americas Tony Andreacchio EMEA Rainer Uhle Related SAP Education Training Opportunities BW360, SAP BW Performance and Administration Related Workshops/Lectures at SAP TechEd 2003 BW354, Performance Tuning for SAP BW, 4hr hands-on Oct. 2, 2003, 11 am, Room H4 SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 47 Questions? Q&A SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 48

25 Feedback Please complete your session evaluation and drop it in the box on your way out. Thank You! The SAP TechEd 03 Basel Team SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 49 Copyright 2003 SAP AG. All Rights Reserved No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, WINDOWS, NT, EXCEL, Word, PowerPoint and SQL Server are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iseries, pseries, xseries, zseries, z/os, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, Informix and Informix Dynamic Server TM are trademarks of IBM Corporation in USA and/or other countries. ORACLE is a registered trademark of ORACLE Corporation. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, the Citrix logo, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, MultiWin and other Citrix product names referenced herein are trademarks of Citrix Systems, Inc. HTML, DHTML, XML, XHTML are trademarks or registered trademarks of W3C, World Wide Web Consortium, Massachusetts Institute of Technology. JAVA is a registered trademark of Sun Microsystems, Inc. JAVASCRIPT is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape. MarketSet and Enterprise Buyer are jointly owned trademarks of SAP AG and Commerce One. SAP, R/3, mysap, mysap.com, xapps, xapp, SAP NetWeaver and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 50

26 SAP TechEd `03 EMEA Online WATCH THE REPLAYS, DOWNLOAD THE SLIDES, AND READ THE TRANSCRIPTS. SAP AG 2003, TechED Basel 2003, BW301_EMEA, Rainer Uhle / 51

SAP BW in a Growing Landscape - Increasing the Benefit of Your BW Implementation - SAP NetWeaver Regional Implementation Group -BI SAP AG

SAP BW in a Growing Landscape - Increasing the Benefit of Your BW Implementation - SAP NetWeaver Regional Implementation Group -BI SAP AG SAP in a Growing Landscape - Increasing the Benefit of Your Implementation - SAP NetWeaver Regional Implementation Group -BI SAP AG Content Overview Content Extend the Reach of Integrate new Non-SAP data