Challenges and Chances of Nearline Storage

Size: px

Start display at page:

Download "Challenges and Chances of Nearline Storage"

Reynard Tucker
6 years ago
Views:

1 Challenges and Chances of Nearline Storage Dr. Michael Hahne Hahne Consulting GmbH June 9, 2010

2 Agenda Challenge: Increasing Data Warehouses Solution: Information Lifecycle Management with SAP NLS Best Practice: Enterprise Data Warehousing Best Practice: NLS for Query Performance Hahne Consulting GmbH Q & A / Further procedure Challenges and Chances of Nearline Storage 2

3 Challenges IT In your opinion, what are the greatest challenges for IT within the next three years? Managing the growing volume of data while having to keep costs low Defining efficient SLAs between IT and business areas Automated storage management Compliance with legal regulatory requirements by IT Business process optimization and management in IT Total 500 to 999 employees 1,000 and more employees Business process optimization and management enterprisewide Increase of IT and data security Very important......unimportant Source: TechConsult GmbH Basis: Total: 187; employees: 85; Lünendonk GmbH 1,000 and more employees: 102 companies Challenges and Chances of Nearline Storage 3

Data Growth Please tell us the projected growth rate of your data volume for the next two years in the following areas (Total) Office documents (incl.

4 Data Growth Please tell us the projected growth rate of your data volume for the next two years in the following areas (Total) Office documents (incl. imaging/ multimedia) Source: TechConsult GmbH Lünendonk GmbH Multiple answers possible Basis: 153 companies Challenges and Chances of Nearline Storage 4

5 Challenge for Data Warehouse With projected compounded annual growth rates for databases exceeding 125%, organizations face two basic options: 1) Continue to grow the infrastructure (e.g., server size, storage capacity) OR 2) Develop processes [and architectures] to separate dormant [archive-ready] data from active data. Meta Group Report Databases on a Diet Challenges and Chances of Nearline Storage 5

6 Typical Data Warehouse Problems We Can t Meet our Batch Windows KPI and report delivery Database backup Reconstruction in Data Warehouse Our Costs are Spiraling Storage / Hardware / Replication Computer center operation (room / energy / air conditioning) Data administration The Targets Keep Changing New business requirements Specific project requirements External requirements / Audit / Internal audit Challenges and Chances of Nearline Storage 6

7 Expenses for Storage (Disks, tape systems, network, administration) grow by 15% to 20% per year even though the storage costs decrease by 30% Ref.: McKinsey Challenges and Chances of Nearline Storage 7

8 Storage Costs Challenges and Chances of Nearline Storage 8

9 Moore s Law, Kryder s Law and the major exception Moore: Duplication of transistors per IC every two years Kryder: Information density on disks doubles every 18 months Factors of Growth: Transistors/Chip: >100,000 since 1971 Disk density (Bit/mm^2): >100,000,000 since 1956 Disk speed: 12.5 since % 40% 35% 30% 25% 20% 15% 10% 5% 0% Compound Annual Growth Rate The disk speed barrier dominates everything! Source: Monash Research Transistors/Chips since 1971 Disk Density since 1956 Disk Speed since Challenges and Chances of Nearline Storage 9

Management Challenges Performance Data Growth What are the

10 Results: Failed Service Levels Performance declines Batch window insufficient WORKLOAD COMPLEXITY Costs Data Management Challenges Performance Data Growth What are the solutions? Challenges and Chances of Nearline Storage 10

11 Information Lifecycle Management Online: Persistent in Data Warehouse Data modelling and multilayered architectures relevant Housekeeping Nearline: Transparent access "BW on a diet" Offline: Classical Archiving Frequently read/updated data Online Database Storage Near Line Storage Data Archiving Infrequently read data Very rarely read data Challenges and Chances of Nearline Storage 11

12 Data Access vs. Data Growth Typical Data Growth Typical Data Access vs. Data Growth With increasing data volume, the frequency of accessing data changes significantly Challenges and Chances of Nearline Storage 12

13 Information as Corporate Asset We Do not Know What we not Know... The Known Current BI implementations are set up to answer known requirements The Unknown Little or nothing is done to be prepared for unpredictable future information needs Challenges and Chances of Nearline Storage 13

14 Data Management Best Practice: Enterprise Data Warehousing M Multi-Cube M Multi-Cube Virtual Reporting Layer Virtual Analytical layer External schema B NLS Partition B NLS Partition B NLS Partition B Basis-Cube B Basis-Cube B Basis-Cube Reporting Layer Analytical layer major aggregated data DSO (st.) DSO (st.) DSO (st.) DSO (st.) DSO (st.) DSO (st.) DSO (st.) DSO (st.) Ready to use enriched data are used to fill Data Marts ~ ¼ to 1/3 of data volume Integration Layer Acquisition Layer Granular, not transformed data 1:1 related to ERP System ~ ¼ to 1/3 of data volume Challenges and Chances of Nearline Storage 14

15 NLS and Multilayered Architectures: NLS and Acquisition Layer Acquisition Layer: complete extraction history Data: granular and not transformed Storage in write-optimized DSOs in general Request-based Archiving after having performed QA of data Archiving within the daily load management Only the last batches of data to be supplied remain online Avoiding new extractions of historical data Challenges and Chances of Nearline Storage 15

16 NLS and Multi-layer Architectures: NLS and Integration Layer Integration Layer: Single Point of Truth Data: granular and transformed Storage usually in standard DSOs Time-slice based archiving according to the Retention Policy for nonchangeable data Data required in load management remain online (avoiding look-ups to nearline data) Flexible adaptation and correction of transformations Challenges and Chances of Nearline Storage 16

17 NLS and Multi-layer Architectures: NLS and Integration Layer Data: aggregated with specific analysis focus General storage in Info Cubes Data Marts on Demand are built flexible on the basis of integration layers Newly aggregated views are constructed as required, based on the Integration Layer Time-slice based archiving according to the Retention Policy if required, monthly, yearly Usually remain online for 2 years B NLS Partition B NLS Partition B NLS Partition B Basis-Cube B Basis-Cube B Basis-Cube Reporting Layer Challenges and Chances of Nearline Storage 17

18 Logical Partitioning Will be unnecessary by NLS along the time axis Regional partitioning (for ex. induced by load management) is kept 2010 Aggregates / Index / Snapshot MP NLS Cubes Challenges and Chances of Nearline Storage 18

19 Data Management Best Practice: Faster Queries with NLS Challenges and Chances of Nearline Storage 19

20 Data Management Best Practice: Faster Queries with NLS Challenges and Chances of Nearline Storage 20

21 NLS accelerates Queries Leaner Basis Cubes are faster NLS partitions on the basis of CBW NLS IQ are usually faster than Basis Cubes Efficient creation of "Snapshots" (100% NLS partitions) also accelerate Challenges and Chances of Nearline Storage 21

Advantages of a Data Management Strategy Performance Faster load times Faster queries Costs Less storage costs Less administration Controlled system growth Availability Data availability: faster

22 Advantages of a Data Management Strategy Performance Faster load times Faster queries Costs Less storage costs Less administration Controlled system growth Availability Data availability: faster rollups and change runs etc. System availability: Less downtime for backups, upgrades etc. Enterprise Data Warehousing "Data Marts on Demand": More flexibility, faster rollout Challenges and Chances of Nearline Storage 22

Efficient BW Data Management Strategy: Key Points 04.

23 Efficient BW Data Management Strategy: Key Points Challenges and Chances of Nearline Storage 26

24 How to get started Analysis Workshop Target: Identify "Quick Wins" ROI in a few months Proof of Concept Test installation Piloting Project and Usage Analysis and design Implementation Operation Health check Challenges and Chances of Nearline Storage 27

25 Support by Hahne Consulting Setup and implementation of a BI strategy Conception of BI Architectures Support in the Business Analysis Conceptional Modelling of BI systems Optimization of BI landscapes with respect to costs and performance Data modelling for BI systems Data Warehouse Modelling Setup of EDW Architectures Implementation of analytical applications on the basis of SAP, SAS, IBM/Cognos, Microsoft Challenges and Chances of Nearline Storage 28

26 Contact Data Dr. Michael Hahne Königsberger Str Bretzenheim Tel Challenges and Chances of Nearline Storage 29

Efficient Data Management with Near-Line Storage

Efficient Data Management with Near-Line Storage The Fast Track to SAP Knowledge Dr. Michael Hahne Product Manager, SAND Technology AGENDA Information Lifecycle Management Near-Line Functionality Enterprise