Operational Hadoop and the Lambda Architecture for Streaming Data

Size: px

Start display at page:

Download "Operational Hadoop and the Lambda Architecture for Streaming Data"

Warren Walsh
6 years ago
Views:

1 Operational Hadoop and the Lambda Architecture for Streaming Data 2015 MapR Technologies 2015 MapR Technologies 1

2 Topics From Batch to Operational Workloads on Hadoop Streaming Data Environments The Lambda Architecture the What and Why Example Deployments 2015 MapR Technologies 2

3 Review: Apache Hadoop Does This Big Compute Task 2015 MapR Technologies 3

4 Scaling on Hadoop Low Data volume, velocity Scale out with commodity hardware High Low Data variety Use the right tool for unstructured, multi-structured, semi-structured, non-relational data High 2015 MapR Technologies 4

5 Scaling on Traditional Technologies Low Data volume, velocity Scale up to bigger, faster machines High Low Data variety Extensive data modeling and ETL High 2015 MapR Technologies 5

6 Batch versus Operational Batch is a built-in delay. Operational is an overloaded term. You might have to wait here for the next window Real-Time? Business operations? Interactive? Industrial operations? In this context: more this than this 2015 MapR Technologies 6

7 Examples of Operational Workloads Immediate reads/writes/updates to live data Immediate access to data Real-time analysis (operational reporting) Let s separate batch from operational in common Hadoop use cases 2015 MapR Technologies 7

8 Common Uses for Hadoop Enterprise Data Hub (or Data Lake ) Anomaly Detection Internet of Things Customer 360 View Fraud Detection Recommendation Engine 2015 MapR Technologies 8

9 Enterprise Data Hub Architecture Enrich data in Hadoop Analyze BI REPORTS AND APPLICATIONS Load more data sources PARSE, PROFILE, ETL CLEANSE, MATCH RELATIONAL, SAAS, MAINFRAME LOAD Graphical User Interfaces DATA MARTS DATA WAREHOUSE DOCUMENTS, S BLOGS, TWEETS, LINK DATA REPLICATE, CDC Batch Processing MR, YARN, Hive, Pig, etc. HBase, other data stores Interactive Querying Drill, Impala, Presto, etc. LOAD LOG FILES, CLICKSTREAMS STREAMING Distributed File System Offload / Enrich / Reload 2015 MapR Technologies 9

10 Example Data Flow within Hadoop Hadoop Canonical, immutable data Transformed data Snapshot data Analyst data 2015 MapR Technologies 10

11 Lambda Architecture ALL DATA BATCH RECOMPUTE PRECOMPUTE VIEWS BATCH LAYER SERVING LAYER NEW DATA STREAM BATCH VIEWS REAL-TIME VIEWS MERGE MERGED VIEW REAL-TIME DATA PROCESS STREAM REAL-TIME INCREMENT INCREMENT VIEWS SPEED LAYER 2015 MapR Technologies 11

12 Why the Lambda Architecture? Explained in How to Beat the CAP Theorem blog by Nathan Marz What is the CAP Theorem? You have to compromise with distributed systems Consistency data in is same as data out Availability the system can always respond with an answer Partition Tolerance the system still runs if nodes lose a connection You get 2 out of 3 (actually, either C or A) The Lambda Architecture can be about more than beating CAP 2015 MapR Technologies 12

13 Key Principles of the Lambda Architecture Store an immutable, constantly growing data set Only add new data points Recompute queries from raw data to avoid complexity Uses incremental algorithms to reduce query latency 2015 MapR Technologies 13

14 Lambda Architecture Batch Layer ALL DATA BATCH RECOMPUTE PRECOMPUTE VIEWS BATCH LAYER NEW DATA STREAM BATCH VIEWS 2015 MapR Technologies 14

15 Lambda Architecture Speed Layer NEW DATA STREAM REAL-TIME VIEWS REAL-TIME DATA PROCESS STREAM REAL-TIME INCREMENT INCREMENT VIEWS SPEED LAYER 2015 MapR Technologies 15

16 Lambda Architecture Serving Layer BATCH VIEWS REAL-TIME VIEWS REAL-TIME DATA MERGE SERVING LAYER MERGED VIEW 2015 MapR Technologies 16

17 Lambda Architecture ALL DATA (HDFS) BATCH RECOMPUTE PRECOMPUTE VIEWS (Hive) HADOOP BATCH LAYER SERVING LAYER NEW DATA STREAM KAFKA BATCH VIEWS REAL-TIME VIEWS REAL-TIME DATA MERGE MERGED VIEW (HBASE) PROCESS STREAM REAL-TIME INCREMENT INCREMENT VIEWS STORM SPEED LAYER 2015 MapR Technologies 17

characteristics Location, time stamp, electrical current, displacement, flow, sediment PPM, etc.

18 Example Use Case Predictive Maintenance Data and use case based on one of our customers Time series sensor data from oil well pumps Transactions and predictive analytics to predict imminent pump failure Data characteristics Location, time stamp, electrical current, displacement, flow, sediment PPM, etc. High volume, high velocity Higher resolution (measurements at a higher frequency) enables more accurate analytics 2015 MapR Technologies 18

field Failed pump measurement pattern Imminently failing

19 Example Use Case Predictive Maintenance (cont.) Sample queries and analysis Dashboard of state of oil field Failed pump measurement pattern Imminently failing measurement pattern Drill-down navigation 2015 MapR Technologies 19

20 What about Historical Analytics on Streaming Data Need static view of data Data can t be moving as you re building your model vs. Need repeatable view of data What if model is wrong? How can you fix it if data has changed? Need comparisons for better models Alternate models should be run on same data Validate legitimacy for future data states Compare to previous state Check out the different snapshot implementations on Hadoop 2015 MapR Technologies 20

21 Managed Security Service Provider Customer data, network security event data Anomaly detection on large volumes of security event data, analytics on customer data to enable incremental sales 2015 MapR Technologies 21

22 Telecommunications Company Customer profile data, customer behavior data Analytics on customer behavior for better recommendations 2015 MapR Technologies 22

23 Q & A Engage with maprtech mapr-technologies MapR dalekim@mapr.com maprtech 2015 MapR Technologies 23

Common Customer Use Cases in FSI

Common Customer Use Cases in FSI 1 Marketing Optimization 2014 2014 MapR MapR Technologies Technologies 2 Fortune 100 Financial Services Company 104M CARD MEMBERS 3 Financial Services: Recommendation Engine