Creating an Enterprise-class Hadoop Platform Joey Jablonski Practice Director, Analytic Services DataDirect Networks, Inc. (DDN)

Size: px

Start display at page:

Download "Creating an Enterprise-class Hadoop Platform Joey Jablonski Practice Director, Analytic Services DataDirect Networks, Inc. (DDN)"

Garey Craig
6 years ago
Views:

1 Creating an Enterprise-class Hadoop Platform Joey Jablonski Practice Director, Analytic Services DataDirect Networks, Inc. (DDN)

2 Who am I? Practice Director, Analytic Services at DataDirect Networks, Inc. 3+ years with Hadoop, 12+ with HPC Contact 2

3 Why Hadoop? Scalable Performance & Capacity Growing Ecosystem (Flexibility) Established APIs & Interfaces Location on the adoption curve Proven base to create Analytical Platforms 3

4 What is Enterprise Class? Scalable OPEX & CAPEX Manageable Integration with existing tools Flexible Workflow Process Integration No Rip & Replace Metrics to manage towards Business Driven, Technological Capabilities 4

5 The Big Data Challenge The Big Data Equation: Volume Velocity Variety + + Petabytes of Data Trillions of Objects GB/s TB/s Millions of IO/s Object Operations Structured Unstructured Streams & Batches

6 Analytics Looking for Actionable Information Billions of Data Points to Consider Consumer purchasing trends Product perception Drug Discovery Genomics Surveillance Financial Analysis

7 Data Gravity Applications DATA Services 7

8 Why is data Analytics so hard? Technical Business Hacking Skills Business Acumen Math & Statistics knowledge Data Science Traditional Research Substantive Expertise Communications Analytics Poor Decisioning Curiosity

9 What is Hadoop missing today? Active-Active high-availability Established management tools Enterprise integration mindset Enterprise class hardware Consistent version-compatibility & deployment Efficient CAPEX & OPEX scaling Resource management/slas/qos Security. 9

10 Hadoop Operational Considerations Deploy Upgrade Manage Respond Monitor Software Platform Hardware Platform

11 Todays Enterprise Picture The Cloud 11

12 Getting there. Improved Results Insight Modify Behavior

13 Hadoop Architectural Considerations 13

14 Planning for Growth Adoption Higher is Better Goal for Human Costs Capacity Performance Scalability User Growth 14

15 Shared v. Commodity Shared Component Approach Lower Operational Costs Efficient operational resource scaling Shared resources with other IT platforms Efficiency in computing, connectivity & service placement Commodity Server Approach Lower Entry Costs Shorter MTBF Inefficient scaling of tools and processes Mis-match with traditional IT operations models 15

16 Ethernet v. Infiniband Infiniband 100% Storage Management Offload End-End InfiniBand Networking with RDMA Acceleration Real-Time Data Delivery to Provide MapReduce Process Consistency Smaller Compute, Compact Storage to Minimize Data Center Impact Ethernet Compatibility, ensured connectivity Limitations in traffic types and bandwidth availability High CPU/Overhead cost Minimal options for offloading with Linux environments 16

17 Analytic User Types Empowered Users Aware Users Enabled Users 17

18 Hadoop Enterprise Integration Monitoring & Response Extract Transform Load APIs Integration Data Information Insight Results 18

19 And finally, Hadoop is more then just hardware, It is about an ecosystem of hardware & software. about integrating with existing systems. a toolkit to build Analytical Platforms. a component of the larger corporate processes and mandates. a component of the wider business KPIs. 19

20 Q&A 20

Intro to Big Data and Hadoop

Intro to Big Data and Hadoop Intro to Big and Hadoop Portions copyright 2001 SAS Institute Inc., Cary, NC, USA. All Rights Reserved. Reproduced with permission of SAS Institute Inc., Cary, NC, USA. SAS Institute Inc. makes no warranties