SAP and Hadoop, better together

Size: px

Start display at page:

Download "SAP and Hadoop, better together"

Gilbert Arnold
5 years ago
Views:

1 SAP and Hadoop, better together A technical deep dive

2 The rising cost of traditional enterprise data warehouse solutions, combined with the fast-growing volumes of data much of it unstructured and the need to store and analyze that data are creating a new IT challenge. It s one that many organizations are attempting to solve by marrying EDW with open-source big data solutions. In some cases, these marriages have created partnerships that are peaceful, productive and mutually beneficial. But far too many have ended in expensive divorce. This paper describes the steps DXC Technology recommends to create data partnerships that are productive and enduring. Optimizing your environment Many organizations have selected an EDW solution based on SAP BW/HANA. While this is a highly structured and organized data warehouse, SAP BW/HANA can be somewhat limited and expensive. To overcome this, smart organizations will undertake an initial assessment to identify potential bottlenecks and suggest candidates for offloading to cheaper, more efficient platforms (Figure 1). DWO workshop DWO assessment DWO roadmap Determine approach Evaluate data architecture Align business strategy to use cases and data architecture Prioritize critical workloads Identify candidates for migration to cloud Define current state/future state Map existing and new workloads - Business priority - Business value - Implementation complexity - Optimization or migration Run diagnostics DWO implementation Evaluate current state and chart future state while identifying transformation tools Identify and prioritize target workloads for optimization or migration Deploy appropriate tools and run POCs to evaluate resource and skills Conduct required assessment, analysis, pilot evaluations and planning Implementation to-be data architecture, including hardware, software and solutions Implement data workload optimization work streams Train resources to manage modernized architecture and optimized workloads Expertise DXC data scientists, technology experts, industry SMEs Use case library Industry and business function examples Guided processes Proven processes to accelerate time to value Big data platforms Hadoop/Spark, Microsoft Azure services, Vertica, etc. Big data infrastructure Azure/AWS/DXC clouds, Dell Platform flexibility On-premises or cloud-based delivery models Data discovery lab Rapid deployment of data discovery labs Figure 1. Data workload optimization (DWO) services 1

3 DXC, in cooperation with our partners, supports an assessment process using both manual and semi-automated tools and techniques (Figure 2). General Performance CPU utilization CPU wait I/O Memory utilization Total disk space Max disk space Query Statistics Top 5 execution duration jobs by utility Load Export Backup Max disk space Transformation Logic DB objects containing transformation logic Views Indexes Triggers Scripts Data Temperature Collect statistics on data temperature and storage to identify data that potentially can be offloaded Figure 2. Assessment building blocks 2

4 SAP BW/HANA optimization patterns Once the initial assessment is completed, it s time to decide which data will be offloaded and how. EDW workload optimization can have different scopes, often depending on the organization s strategy and degree of cloud readiness. To help organize the process of adopting big data analytics, DXC offers codified optimization patterns (Figure 3). Organizations can use these optimization patterns to either proceed on their own or let DXC guide them on where to begin and how to proceed. Augmentation Partial migration Area 1 HOT COLD Area 2 New Sources File Stream Big Data Platform New data sources (e.g. semi-structured) New transformation approach (e.g. stream) New analytics possibilities (e.g. deep learning) Big Data Platform Cold data offload (e.g. history) Heavy area offload (detailed data in big data, aggregates in EDW) replacement EDW Integration Full replacement Big Data Platform Area 1 HOT COLD Area 1 Big Data Platform Two different EDW need to be integrated (company merger or reorganization) Different technology Different data model Leaving current tool (database forklift) Capturing data directly from the source ( replacement) Figure 3. Optimization patterns 3

5 Augmentation The most popular pattern for a starting point is augmentation, as illustrated in Figure 4. In this pattern, the existing analytics fabric stays unchanged, but it s surrounded by improved technology to enhance the functionality. This is done by: Adding new data sources such as semi-structured or unstructured data Introducing additional data-transformation approaches, such as stream processing Providing new data mining and machine learning algorithms, such as deep learning, as well as other possibilities that include online scoring and learning models on the full data set Build in BW or SAP BODS or other tool SDA HANA view SAP BO/SAP Lumira or other front-end tool New Sources File Stream SAP BW/HANA VORA JDBC to HIVE/ Impala/ Spark Analytics Platform Figure 4. Augmentation 4

6 Augmentation of an existing EDW solution assumes that the analytics platform adds new features. In this case, no changes are needed in the source, data warehouse or front end. The DXC Analytics Platform, combined with the less costly storage, provides plenty of modern analytics functionalities, including: New data sources (e.g., semi-structured) New transformation approaches (e.g., stream, schema on read) New analytics possibilities (e.g., deep learning, online scoring) Under the augmentation approach, the fastest way to provide data for users is to connect reporting applications to the analytics platform via JDBC/ODBC drivers. This can be done long before setting up connections between SAP and Hadoop. Users can still work in SAP BusinessObjects (BO), SAP Lumira or other reporting tools, but now they ll also see external data collected in the analytics platform. It s also possible as part of this scenario to leverage SAP-to-Hadoop integration options. These include smart data access (SDA), used to first gain access to data collected in the analytics platform through HANA Views, then provide it to existing reporting tools; and SAP Vora, an in-memory query engine used to optimize data transformation with Apache Spark integration. Partial migration The next-most popular pattern is partial migration. This pattern includes levels of migration that range from the simple offloading of cold data (archival data that is rarely or never accessed by users) to the complex migration of extract, transform and load () processing and the staging layer to the DXC Analytics Platform. This then loads back to the existing EDW data that s now ready for reporting or analysis. The partial migration pattern not only provides the same features offered in augmentation, but also gives organizations an opportunity to reduce their analytics costs. It does this by reducing the solution s size and complexity. Partial migration options include: Cold data migration, with an option to load back or access if needed and stage migration, where data ready for reporting goes to SAP, making this a good option for non-sap data sources Heavy area migration, which is best for big tables, resource-consuming aggregation or other transformation; summary data or aggregates may be loaded back to the EDW Minimizing SAP BW/HANA, keeping it only for regulatory or standard reporting and using the analytics platform for other activities such as ad-hoc querying, data discovery, machine learning and dashboards 5

7 Build in BW or SAP BODS or other tool SDA HANA view SAP BO/SAP Lumira or other front-end tool SAP BW/HANA Area 1 Area 2 HOT COLD Sqoop NLS JDBC to HIVE/ Impala/ Spark VORA Analytics Platform Figure 5. Cold data migration Partial migration cold data The first step in this scenario is to assess which part of the EDW is cold that is, containing data that s either never or only rarely used. This assessment can be done either manually by skilled database administrators or by using automated assessment tools. After cold data is identified, it can be offloaded to the analytics platform (Figure 5) using SAP near line storage (NLS) or Apache Sqoop, an open-source tool for transferring bulk data between structured databases and those based on Hadoop. This offloading process should be repeated on a regular basis daily, weekly or monthly, depending on the organization s needs and resources. Gaining access to the offloaded cold data can be handled in any of several ways: Leverage SDA through HANA Views (and Vora to optimize data transformation) Use Sqoop to load back the data needed for analysis Use direct integration with the analytics platform via JDBC connectors to SAP BO or other reporting tools Use the analytics platform s reporting tools, (e.g., Ambari Views, Zeppelin, Spotfire, Jupyter) 6

8 Partial migration and stage This is usually (but not only) done in an EDW, if most of the data transformation takes place in the database stage area. In fact, the transformation workload can consume up to 50 percent of all EDW resources. For this reason, in the and stage offloading scenario, the analytics platform is used to keep all stage data in Hadoop Hive (a data warehouse system), and then to transform data to the final schema using the power of Spark. Finally, data ready for reporting can be loaded to the EDW using Sqoop or SAP HANA Smart Data Integration (SDI), as illustrated in Figure 6. This is an especially good approach for non-sap data sources. In this type of partial migration, the DXC Analytics Platform can transform essentially all types of data, including: Structured data Semi-structured data (e.g., logs and clickstreams) Nanostructured data (e.g., text and pictures) Batch and stream data Small portions with huge velocities or big files/tables Build in BW SAP BO/SAP Lumira or other front-end tool Non-SAP Sources SAP Sources Area 1 SAP BW/HANA Sqoop Area 2 SDI JDBC to HIVE/ Impala/ Spark : Spark/HDF Transform Spark Stage: Hive Analytics Platform Figure 6. and stage migration 7

9 Partial migration heavy area In some cases, the EDW is used to transform and store detailed data from operational systems that involve huge amounts of data. This could include data for telco service usage, bank operations, retail sales, e-commerce clickstreams, online advertising conversions and internet of things data. In these cases, the analytics platform is the best place to reroute such data. This can be done by either using existing tools or connecting the analytics platform directly to the source (see Figure 7). Once this is done, the analytics platform will efficiently transform and aggregate these data items in a format ready for final reporting. Aggregated (and much smaller) data can be loaded back as in other previous scenarios. But it s even better to leave that data in its repository. Then access for users can be provided by: Leveraging SDA through HANA Views (also Vora to optimize data transformation) Using direct integration to the analytics platform via JDBC connectors to SAP BO or other tools Using the analytics platform s reporting tools (e.g., Ambari Views, Zeppelin, Spotfire) SAP BODS or other tool SDA HANA View SAP BO/SAP Lumira or other front-end tool OR : Spark/HDF Area 1 SAP BW/HANA Area 2 Transform Spark Area 2 Hive VORA Analytics Platform JDBC to HIVE/ Impala/ Spark Figure 7. Heavy area migration 8

10 Partial migration minimizing SAP BW/HANA This approach to optimizing SAP BW/HANA s size and cost focuses on business perspectives. The big question is: What needs to stay on SAP BW/HANA? These areas might include: Regulatory reporting Financial reporting Internal group or branch reporting Predefined strict or structured reporting Everything else including advanced analytics, machine learning, process optimization, data wrangling and discovery, ad hoc reporting and dashboards can be moved to the cheaper and faster DXC Analytics Platform (as shown in Figure 8). This can also leverage the power of based on Spark. Build in BW or SAP BODS SAP BO/SAP Lumira or other front-end tool Non-SAP Sources OR Area 1 OR SAP BW/HANA Sqoop Area 2 JDBC to HIVE/ Impala/ Spark : Spark/HDF Analytics Platform Figure 8. Minimizing SAP BW/HANA 9

11 Other less-popular patterns Although most organizations start with the augmentation pattern and then progress through several variants of partial migration, some organizations need to go further. For them, the best option is the full replacement pattern. This pattern assumes that a traditional analytics solution will be turned off. Obviously, the migration project to turn off the traditional solution needs to go through phases, migrating step by step. But the final architecture will need to be optimized for full replacement. Similarly, optimization of the architecture will need to focus on the needs of the non-edw components, including, business intelligence (BI) and machine learning. Full replacement is the most challenging pattern from a change-adoption perspective, but it also provides the biggest opportunity for reducing cost and complexity. It does so mainly by removing the need for expensive enterprise software licenses. In addition, full replacement empowers the organization to focus on the business outcomes of analytics, rather than on their technical complexities and maintenance requirements. Organizations that opt for full replacement have several options: For, it s possible to keep the existing source, and either reroute its data flows to the analytics platform or connect the analytics platform to every data source and use native tools such as Sqoop, Spark and NiFi. Existing front-end reporting tools may either be integrated with the analytics platform via JDBC/ODBC connectors or replaced by analytics platform reporting tools. Full replacement programs can be rolled out in phases. In this way, the EDW architecture will temporarily pass through partial migration scenarios for improved safety and reliability. Organizations may also want to consider an EDW integration pattern following a major merger or corporate reorganization. In these situations, the organization may find itself with two or more data warehouse, business intelligence or analytics solutions. An EDW integration pattern assumes that the complexity of two or more systems for analytics is straightforward, hidden by a modern approach using solutions such as Schema on Read and Document Databases. As a result, the end user should see a single system view even if the underlying data is not integrated. This pattern provides a quick improvement and gives the organization additional time to analyze and develop the final architecture; for example, an eventual partial migration or full replacement. However, because this pattern also adds both an additional layer and more complexity to the architecture, it should be treated as a temporary solution only. 10

12 Migration patterns as a step from augmentation to full replacement Choosing a migration pattern is not a one size fits all decision. On the contrary, no single solution will likely prove sufficient for every organization (see Figure 9). That said, one recommended best practice is to start small while also creating enough agility to grow quickly in many directions. The larger goal is to use the right methodology and approach for the unique needs of the business. Project size Small Medium Large Partial migration Duration (months) Team size (FTE) Augmentation Area 1 Area 2 HOT COLD Front End Full replacement Front End Big Data Platform S M L Area 1 Area 1 HOT COLD Front End New Sources File Stream Big Data Platform Big Data Platform Pattern/project size Small Medium Large Augmentation New batch data source without integration with EDW Streaming processing with lookups to EDW High volume, velocity and variety of new data Partial migration Cold data offload without changes in and reporting Heavy area offload aggregated data may return to EDW replacement offload of and stage area Full replacement Replacement of small EDW without replacement Full replacement, including Figure 9. DWO patterns adaptive process 11

13 Client examples A global consumer packaged-goods supplier recently implemented the augmentation pattern with help from DXC. The company s end users received quick and easy access to external data located on the DXC Analytics Platform, including information related to macroeconomics, market research, brand health, surveys, media and shoppers. They also received large amounts of operational data. All of this data was analyzed with SAP Lumira data visualization and alternative data-discovery tools. As a result of its augmentation implementation, the consumer goods company: Derived business value from the capability to analyze between different functional data sets Gained new analytical insights by analyzing ad-hoc and operationalized data sets Extended the retention period of large-scale data sets in an economically viable way Used a platform that supports the modification of the data schema in a way that doesn t require the repopulation of the database or the reorganization of data at the data-lake storage layer Similarly, a major energy company, also with help from DXC, recently completed the partial migration pattern. As part of the project, DXC offloaded cold data from the company s SAP HANA implementation to a cloud-based solution. As a result, the energy company: Reduced its costs and improved its CAPEX-to-OPEX ratio Optimized cost of cloud services via the selection of services and instances DWO (such as auto-scale for compute) Optimized data models for better processing and consumption Improved processing on the platform using Spark and HQL Achieved an iterative deployment to the production database Improved ongoing performance tuning in the production database Another company, this one in the telecommunications industry, turned to DXC for help with several EDW-related challenges. The company sought to lower its operational costs for SAP HANA and non-sap business applications. It also wanted to deploy advanced analytics on its structured and unstructured data. The company wanted to improve information sharing with internal and external stakeholders. And it wanted to improve access to non-sap data. Using the DXC Analytics Platform, the telecom company cut its operational costs by 20 percent from baseline. It also gained scalability and agility in the EDW analytics area, enabling the company to set up predictive analytics on both hot and cold data. In addition, the company has democratized access to non-sap data by storing it on a new landing zone. Finally, a manufacturing company worked with DXC and our partner Datavard to achieve meaningful reductions in both the resources and costs for SAP HANA. Our assessment step obtained a 20 percent reduction of required memory (in the InfoCubes area) and a 50 percent reduction of resources needed in the decision support objects (DSO) layer. Also as a result of our refinements, the company was able to safely offload 3 terabytes of data to NLS, further lowering its operational costs. 12

DXC, your trusted partner DXC specializes in developing workload-specific transformation and migration strategies aligned with your business objectives.

14 DXC, your trusted partner DXC specializes in developing workload-specific transformation and migration strategies aligned with your business objectives. We offer a broad set of migration capabilities to support a diverse array of technologies, geographies, regulatory requirements, operating models and target environments. The DXC Analytics Platform, being infrastructure-agnostic, makes it easy and efficient to follow the best suitable migration path for your enterprise data warehouse. DXC has experience in managing enterprise hybrid environments, a balanced knowledge of traditional and next-generation infrastructure, and an offering of comprehensive services. All this enables us to successfully deliver numerous migration projects to all types of cloud platforms and on-premises solutions. We re eager to help your business create a happy marriage between your EDW and big data. About the authors Sławomir Folwarski is the senior architect, DXC Analytics Platform, focused on data workload optimization and big data platform architecture. He has 17 years of experience in the telco, public sector, automotive, logistics and finance industries with expertise in data warehousing, business intelligence and Hadoop technologies. Sławomir has a double master s degree in IT and economics and keeps himself updated by taking online training courses from Coursera, edx and Udacity. He is also certified in Architecting Microsoft Azure Solutions and in Agile PM and PMP. Learn more at analytics About DXC Technology DXC Technology (DXC: NYSE) is the world s leading independent, end-to-end IT services company, serving nearly 6,000 private and public-sector clients from a diverse array of industries across 70 countries. The company s technology independence, global talent and extensive partner network deliver transformative digital offerings and solutions that help clients harness the power of innovation to thrive on change. DXC Technology is recognized among the best corporate citizens globally. For more information, visit dxc.technology DXC Technology Company. All rights reserved. MD_8142a-18. April 2018

SAP Cloud Platform Big Data Services EXTERNAL. SAP Cloud Platform Big Data Services From Data to Insight

SAP Cloud Platform Big Data Services EXTERNAL. SAP Cloud Platform Big Data Services From Data to Insight EXTERNAL FULL-SERVICE BIG DATA IN THE CLOUD, a fully managed Apache Hadoop and Apache Spark cloud offering, form the cornerstone of many successful Big Data implementations. Enterprises harness the performance