SAS & HADOOP ANALYTICS ON BIG DATA

Size: px

Start display at page:

Download "SAS & HADOOP ANALYTICS ON BIG DATA"

Dennis Malone
6 years ago
Views:

1 SAS & HADOOP ANALYTICS ON BIG DATA

2 WHY HADOOP? OPEN SOURCE MASSIVE SCALE FAST PROCESSING COMMODITY COMPUTING DATA REDUNDANCY DISTRIBUTED

3 WHY HADOOP? Hadoop will soon become a replacement complement to: Business Intelligence; Data Warehousing; Data Integration; Analytics. HADOOP IN PRODUCTION: YES 10% NEVER #1 reason to go for Hadoop: Analytics (71%) < 12 MONTHS Challenges to Hadoop adoption: Hadoop has no analytic functions built in Cost: hefty payroll due to intensive hand coding 3+ YEARS < 36 MONTHS < 24 MONTHS SOURCE: 10 Myths About Hadoop - TDWI Best Practices Report

4 WHY SAS? ANALYTICS IN-MEMORY HIGH-PERFORMANCE DATA MANAGEMENT BUSINESS INTELLIGENCE DATA VISUALIZATION

5 WHY SAS? ANALYTICAL DECISION MAKING Competitive Advantage Optimize What is the best that can happen? Differentiators Predict What will happen next? Predict Prescribe Optimize What if these trends continue? Why is this happening? Statistical Analysis Forecast Alerts What actions are needed? Raw data Clean data Standard reports Ad hoc reports Query drill down Degree of Intelligence Where exactly is the problem? How many, how often, where? What happened?

6 AN ERA OF ABUNDANCE BIG DATA BIG DATA

7 AN ERA OF ABUNDANCE HADOOP BIG DATA HADOOP

8 AN ERA OF ABUNDANCE ANALYTICS BIG DATA HADOOP ANALYTICS

9 AN ERA OF ABUNDANCE WHERE WE ARE NOW BIG DATA Lots of data HADOOP Processing Power ANALYTICS Intelligence

Scalability: provide comprehensive support from data-to-decision to maximize the value of Hadoop across the enterprise.

10 SAS & HADOOP THE BUSINESS REASONING What organizations are looking for: Accuracy: bring superior analytics to Hadoop for more precise insights. Scalability: provide comprehensive support from data-to-decision to maximize the value of Hadoop across the enterprise. Governance: integrate and manage data in order to promote broad reuse and to comply with IT policies and procedures. Economics: drive bottom-line benefits by boosting the value of analytics infrastructure while reducing TCO.

11 SAS & HADOOP WHY THE MARRIAGE? High-performance Advanced Analytics; Business Intelligence and Data Visualization; At Massive Scale, on Distributed, Commodity Hardware

12 SAS & HADOOP HOW? SAS & Hadoop intersect in many ways: SAS can treat Hadoop just as any other data source, pulling data FROM Hadoop, when it is most convenient; SAS can work WITH Hadoop, lifting data in a purpose-built advanced analytics in-memory environment; SAS can work directly IN Hadoop, leveraging the distributed processing capabilities of Hadoop.

Bridge to traditional SAS environments Hadoop treated as just another data source

13 SAS & HADOOP SAS FROM HADOOP SAS accesses and extracts data from Hadoop to a SAS server for processing, and writes results back. Bridge to traditional SAS environments Hadoop treated as just another data source Performance limited to single pipe bandwidth Ideal when not all data is to be found in Hadoop, or when established process cannot run in Hadoop DATA MOVEMENT

SAS & HADOOP SAS WITH HADOOP SAS accesses and

keeping the data and computations massively

Provides capabilities Hadoop cannot do well

Allows the scaling of data storage and analytics

14 SAS & HADOOP SAS WITH HADOOP SAS accesses and processes Hadoop data on SAS Servers while keeping the data and computations massively parallel. Provides capabilities Hadoop cannot do well Supports advanced analytics via shared computing Allows the scaling of data storage and analytics separately Ideal when analytical rigor, sophistication and governance are required DATA LIFT INTO MEMORY

15 SAS & HADOOP SAS IN HADOOP SAS processes data directly in the Hadoop cluster. SAS Embedded Process enables scalable SAS compute in Hadoop SAS compute is orchestrated via Hadoop technology Data manipulation, data quality, and scoring support Ideal when all data is landing in Hadoop, and Hadoop is the proper place for processing SAS LOGIC

16 SAS & HADOOP SAS IN HADOOP SAS processes data directly in the Hadoop cluster. SAS Embedded Process provides scalable SAS compute in Hadoop SAS compute is orchestrated via Hadoop technology Data manipulation, data quality, and scoring support Ideal when all data is landing in Hadoop, and Hadoop is the proper place for processing

SAS & HADOOP Prepare data IN Hadoop for analytics THE

environment Deploy and manage model score code IN Hadoop Lift

17 SAS & HADOOP Prepare data IN Hadoop for analytics THE PRAGMATIC APPROACH Move data FROM Hadoop into a SAS environment Deploy and manage model score code IN Hadoop Lift data IN to memory for analytics at scale Use the right approach for what needs to be done! Explore data at scale, inmemory WITH data visualization Model data at scale inmemory WITH advanced modeling tools

18 SAS & HADOOP KEY POINTS SAS is the only vendor to work from + with + in Hadoop throughout the analytics lifecycle. All three approaches can be combined and coordinated, complementing each other for each situation. Each approach can evolve, mature and/or morph into the other. Metadata management across the whole analytics life cycle, crossing all Hadoop interactions, is key to success. SAS can help realize the value of Hadoop; bring production-analytics to the platform.

ROGERS MEDIA Data visualization and high performance analytics Processing data on 12 million customers 40 million records per month in Hortonworks More than 600 relevant web

19 ROGERS MEDIA Data visualization and high performance analytics Processing data on 12 million customers 40 million records per month in Hortonworks More than 600 relevant web characteristics Several of us from Rogers in the room looked at each other, and said That is really wicked; that s cool. Chris Dingle Senior Director of Audience Solutions Rogers Communications

.. they can look at data and spend more time analyzing it and become internal

20 MACY S 20% reduction in churn $500,000 annual savings Customer lifetime value analysis More accurate response prediction Optimized promotions... they can look at data and spend more time analyzing it and become internal consultants who provide more of the insight behind the data. Kerem Tomak Vice President of Analytics

to Hive Filename support Execute Pig Scripts and MapReduce

21 SAS/ACCESS TO HADOOP Uses Existing SAS Interfaces Standard Libname syntax PROC HADOOP Datastep and Proc SQL translated to Hive Filename support Execute Pig Scripts and MapReduce Push-down of certain procedures Custom SerDe support SPDE formats

(HDFS) Optimized for interactive queries Similar to Hive in

22 SAS/ACCESS TO IMPALA Massively Parallel Processing (MPP) query engine SQL queries against the Hadoop file system (HDFS) Optimized for interactive queries Similar to Hive in function, but different in implementation Extraordinary performance

23 SAS/ACCESS TO HAWQ Direct, transparent access to the Pivotal HAWQ SQL engine SQL pass-through Enable you to interact with HBase Bulk loading, faster than inserting

24 SAS VISUAL ANALYTICS - EXPLORER Data exploration at massive scale Intuitive visual analytics

25 SAS VISUAL STATISTICS Descriptive and Predictive Modeling Model comparison Dynamic groupby processing

26 SAS IN-MEMORY STATISTICS FOR HADOOP In-Memory Statistics for Hadoop: Interactive Programming interface for SAS model development

27 SAS VISUAL ANALYTICS REPORT DESIGNER Visual Analytic Designer and Viewer: Reporting and analysis for broad audiences

28 SAS VISUAL ANALYTICS VIEWER FOR MOBILE Mobile BI for reporting

29 SAS HIGH-PERFORMANCE DATA MINING Highperformance procedure nodes in SAS Enterprise Miner

30 SAS DATA LOADER FOR HADOOP SAS Code Accelerator (DS2) Embedded Process and Hive Parallel data loading No data movement Data Profiling Data Quality Accelerator

31 SAS DATA MANAGEMENT CAN USE ALL THREE EP EP EP

32 SAS 9.4 SUPPORTED HADOOP DISTRIBUTION Cloudera CDH ( 支持 Kerberos) Hortonworks HDP ( 支持 Kerberos) MapR ( 未验证 Kerberos) Pivotal HD ( 支持 Kerberos) IBM BigInsights ( 未验证 Kerberos)

SAS and Hadoop Technology: Overview

SAS and Hadoop Technology: Overview SAS Documentation September 19, 2017 The correct bibliographic citation for this manual is as follows: SAS Institute Inc. 2015. SAS and Hadoop Technology: Overview.