Big Data and Real Time Analytics Streams and Hadoop

Transcription

1 Big Data and Real Time Analytics Streams and Hadoop Infrastructure Matters 2014 Briefing 2014 IBM Corporation

2 Big Data is more than just Hadoop What can you tell me about Big Data? I want to know all about Hadoop. Big Data is a lot more than Hadoop! And our competitors don t understand this they cannot deliver value on an entire set of Big Data use cases. Service Oriented Finance CMO IBM Big Data and Real Time Analytics 2

3 IBM Big Data Solutions All Data New/Enhanced Applications Information ingestion and operational information zone DB2 BLU Real-time analytics zone InfoSphere Streams Exploration, landing and archive zone BigInsights and Hadoop Information governance zone Enterprise warehouse data mart and analytic appliances zone DB2 BLU and PureData System for Analytics InfoSphere Server and DataStage Cognos Why did it happen? Reporting, analysis, content analytics InfoSphere Discovery What is happening? Discovery and exploration Cognitive Fabric SPSS What could happen? Predictive analytics and modeling What action should I take? Decision management Systems Security Storage On premise, Cloud, As a service Analyze all data, from any source, with the right technology IBM Big Data & Analytics Infrastructure Big Data and Real Time Analytics 3

4 There are two main types of Big Data Real-time Analytics Zone Stream Computing Landing and Analytics Zone Hadoop System Data in motion Data typically not stored Tremendous velocity Ultra low latency required Multiple data sources Huge volumes of unstructured data Data at rest Data stored on disk Huge volumes of unstructured data No pre-defined schemas Too large for traditional tools to process in a timely manner Our competitors do not address both of these! Big Data and Real Time Analytics 4

5 New programming models and low cost hardware solve Big Data problems Streaming Application Streaming Cluster Streaming and Apache Hadoop applications Proven frameworks to process large amounts of data Streaming for data in motion, Hadoop for data at rest Enable applications to transparently work with large clusters of nodes in parallel Hadoop Cluster Clusters of low cost POWER8 servers are ideal for Hadoop and streaming applications Big Data and Real Time Analytics 5

6 Service Oriented Finance wants to gain a competitive advantage from Big Data This application will give our market managers a real advantage! Service Oriented Finance wants to deploy a stock trading application with the following requirements Process millions of trades per second Application must scale Constant flow of input data Microsecond latency Unstructured trade data input Sophisticated analytics logic Service Oriented Finance Market Manager Big Data and Real Time Analytics 6

7 InfoSphere Streams is a platform for Real-time Analytics on Big Data in motion Just in time decisions InfoSphere Streams can meet these requirements! Powerful Analytics Millions of events per second Microsecond Latency Streams is a platform for real-time analytics on Big Data Our competitors do not have this capability Sensor, video, audio, text and relational data sources Big Data and Real Time Analytics 7

8 Streams Makes it easy for data in motion programming Developer Role Eclipse based tools Visual application monitoring Built in accelerators Administrator Role Visual application monitoring Stream data visualization Start/stop jobs Business User Role Visual application monitoring Stream data visualization InfoSphere Streams Console Big Data and Real Time Analytics 8

9 Streams programming is drag and drop simple Source Adapters Operator Repository Sink Adapters Application composition (Optimized Compilation) Big Data and Real Time Analytics 9

10 Streams Studio provides a rich set of Eclipse based tools Drag and drop simple Big Data and Real Time Analytics 10

11 Streams provides more throughput than Apache Storm in analysis benchmark Throughput ( s/s) Throughput - Four Nodes Streams 2.6x-12.3x More Throughput Storm x4 (100%) x8 (100%) x8 (200%) x8 (400%) Parallelism (dataset) Based on IBM internal tests comparing InfoSphere Streams against Apache Storm. Results may not be typical and will vary based on actual workload, configuration, applications and other variables in a production environment. Users of this document should verify the applicable data for their specific environment. Contact IBM and see what we can do for you. Big Data and Real Time Analytics 11

12 InfoSphere BigInsights includes Apache Hadoop to process data at rest Hadoop Cluster Processing Input Storage MapReduce Java Program Result Comprised of a cluster of inexpensive hardware Nodes have processors, memory and disks Special file system Hadoop Distributed File System (HDFS) Special programming model MapReduce Big Data and Real Time Analytics 12

13 The Hadoop Distributed File System (HDFS) distributes data across a Hadoop cluster inputfile.txt B1 B2 B3 B = block R = replica Hadoop Distributed File System B1 R3 B2 R1 R3 R2 B3 R2 R1 Node 1 Node 2 Node 3 Node n A distributed file system that spans all the nodes in a Hadoop cluster Files are split automatically at load time into blocks and spread among Data Nodes System assumes nodes will fail Achieves reliability by replicating data across multiple nodes Elastically scalable Big Data and Real Time Analytics 13

14 The MapReduce framework sends programs out to the data MapReduce Job Map and Reduce Tasks Map and Reduce Tasks Map and Reduce Tasks Map and Reduce Tasks HDFS HDFS HDFS HDFS Node 1 Node 2 Node 3 Node n MapReduce job is sent out to each node Map and Reduce tasks run in parallel across nodes Hadoop framework does a lot of the heavy lifting e.g., moving data between map and reduce tasks Big Data and Real Time Analytics 14

15 BigInsights makes it easy for all Big Data roles Developer Role Eclipse based tooling Read/write access to HDFS Extensive views of jobs and workflows in system Application staging, launch and scheduling center Many built in accelerators InfoSphere BigInsights Console Administrator Role Complete management of cluster Monitor/start/stop components Add/remove nodes Portal style dashboards Business User Role No Java required Spreadsheet tooling Visualization Big Data and Real Time Analytics 15

16 Service Oriented Finance wants to analyze customer complaints We need to know what our customers are complaining about. We can help you do that with sentiment analysis using BigInsights Service Oriented Finance CMO IBM Big Data and Real Time Analytics 16

17 Sentiment Analysis - A Big Data challenge but also a Big Data opportunity Trying to determine Product demand Feelings - Attitudes Emotions - Opinions Thoughts - Desires New product acceptance Competitive threats Threats to brand reputations Huge volumes of unstructured data Advertisement targets Finding sentiment from social media data Big Data and Real Time Analytics 17

18 DEMO: Using BigInsights to Analyze negative sentiment on Twitter I don t trust the web site for on-line banking The service reps are very nice and helpful I love the check guard feature! The ATM fees are ridiculous! Data Source Twitter Topic Service Oriented Finance Likes Love the check guard feature Like the on-line bill pay feature Like that the ATMs are located all over the city Like the service representatives Dislikes Don t trust the on-line banking feature Don t like to wait in line for a long time Don t like the ATM fees Hate the overdraft fees Big Data and Real Time Analytics 18

19 Architecture matters when you design a micro processor for emerging big workloads It s not about the number of transistors, it is what you do with them to handle Big Workloads POWER7 to POWER8 1.2 Billion Transistors 567 mm 45nm to 22nm 4.2 Billion Transistors 650 mm 2 Westmere EX to Ivy Bridge EX 2.6 Billion Transistors 513 mm 32nm to 22nm 4.3 Billion Transistors 541 mm 2 CAPI: Coherent Accelerator Processor Interface POWER8 vs. Ivy Bridge EX - 96 threads/socket vs. 30-4x Memory Bandwidth - 3x on-die Cache - Cache latency reduced by 50% - 5x I/O Bandwidth - 15 metal layers vs. 9 - edram vs SRAM POWER8 Unique Technology - CAPI Technology - Integrated PCIe - Transactional Memory - L4 Cache - Dynamic Overclocking Big Data and Real Time Analytics 19

20 Is Ivy Bridge a new breakthrough architecture? The Ivy Bridge CPU micro-architecture is a shrink from Sandy Bridge Ivy Bridge is a tick in the tick/tock Intel release cycle compared to Sandy Bridge A tick means same architecture as previous with some minor improvements The major improvement is the 22 nm Tri-gate transistor technology from the prior 32 nm technology More transistors More cores, sockets 20% more L3 cache Does the 22nm technology result in better performance? nm = nanometer which is a measure of the CMOS semiconductor device fabrication process. Smaller nm process allows for more transistors Source for Tick Tock Model: Big Data and Real Time Analytics 20

21 Intel s performance per Core is not increasing over previous generation RPE2 per Core Socket HP Servers Sandy Bridge EP 2.9 GHz 16 cores Ivy Bridge EP 2.7 GHz 24 cores Ivy Bridge EX 2.8 GHz 30 cores 0 The number shown is best in each category (sockets and number of cores) Source of RPE2: (Gartner) RPE2 numbers are derived from the following six benchmark inputs: SAP SD Two-Tier, TPC-C, TPC-H, SPECjbb2006 and two SPEC CPU2006 components The data in this tool is derived from RPE2 from Ideas International. Ideas International was acquired by Gartner, Inc. in Gartner, Inc. and/or its affiliates. All rights reserved. Big Data and Real Time Analytics 21

22 The new POWER8 scale-out servers innovation to put data to work POWER8 roll-out is leading with scale-out (1 and 2 Socket) systems Expanded Linux focus: Ubuntu, KVM, and OpenStack OpenPOWER Innovations 1 & 2 Socket Power Systems S812L S822L S822 S814 S824L S824 1-socket, 2U Up to 12 cores 512 GB Memory 6 PCIe Gen 3 Linux only PowerVM & PowerKVM August 29, socket, 2U Up to 24 cores 1 TB memory 9 PCIe Gen3 Linux only PowerVM & PowerKVM June 10, socket, 2U Up to 20 cores 1 TB memory 9 PCIe Gen 3 AIX & Linux PowerVM June 10, socket, 4U Up to 8 cores 512 GB memory 7 PCIe Gen 3 AIX, IBM i, Linux PowerVM June 10, socket, 4U Up to 24 cores Linux only NVIDIA GPU 2H socket, 4U Up to 24 cores 1 TB memory 11 PCIe Gen 3 AIX, IBM i, Linux PowerVM June 10, 2014 Scale-out Big Data and Real Time Analytics 22

23 Power S822L servers are priced competitively to Intel Ivy Bridge servers Comparable TCA Linux on Intel Ivy Bridge + KVM vs. Linux on POWER8 + KVM Server list price* -3-year warranty, on-site Dell PowerEdge R720 HP ProLiant DL380 G8 IBM Power S822L $21,300 $22,763 $22,382 $12,605 $14,068 $14,895 Virtualization - 2 sockets, 3 yr. 9x5 sub./supp. $2,998 KVM for Red Hat on x86 (RHEV) $ 2,998 KVM for Red Hat on x86 (RHEV) $2,998 KVM for Linux on Power (PowerKVM) Linux OS list price - RHEL, 2 sockets, unlimited guests, 9x5, 3 yr. sub./ supp. $5,697 Red Hat subscription and Red Hat support $5,697 Red Hat subscription and Red Hat support $4,489 Red Hat subscription and IBM support Total list price: (Total cost of acquisition) $21,300 $22,763 $22,382 Server model Dell R720 HP Proliant DL380p G8 IBM Power 822L Processor / cores Two 2.7 GHz, E5-2697, Ivy Bridge, 12-core processors Two 3.4 GHz POWER8, 10-core Configuration 64 GB memory, 2 x 300GB 15k HDD, 10 Gb two port Same memory, HDD, NIC * Based on US pricing for Power S822L announcing on April 28, 2014 matching configuration table above. Source: hp.com, dell.com, vmware.com Big Data and Real Time Analytics 23

24 POWER architecture is ideal for Hadoop and Streams workloads Mutli-Threading High I/O Performance Processor and Cache Java Optimization Twice the thread capability to use to run parallel Java workloads More threads enables faster BLU Acceleration Single- Instruction, Multiple Data (SIMD) vector processing More threads enable faster Hadoop workload processing High memory and I/O bandwidth enables faster Cognos ad hoc queries and better SPSS scoring throughput High I/O performance enables Cognos faster ad hoc queries and better SPSS scoring throughput Higher speed that significantly outperforms competitive x86 Hadoop configurations Large processor cache per core, large numbers of cores and high DRAM capacity on a single server benefits Cognos Cubes performance Extensive Soft-Error Recovery, self-healing for solid faults, and alternate processor recovery is ideal for Hadoop-based workloads IBM s Java Virtual Machine is specifically optimized for the POWER architecture to deliver optimal performance of big data and analytics solutions Big Data and Real Time Analytics 24

25 BigInsights on POWER beats the competition with TeraSort benchmark GB/core/min Normalized Performance GB / Core / Min 1.7X Competitor 0.73 Competitor 0.75 Competitor POWER7+ with BigInsights POWER8 with BigInsights 0 Xeon E5-2697v2 8 nodes 192 cores 1 TB Xeon E5-2630v2 24 nodes 288 cores 10 TB Xeon E nodes 256 cores 10 TB POWER7+ 7R2 10 nodes 120 cores 1 TB POWER8 S822L 8 nodes 192 cores 10 TB Big Data and Real Time Analytics 25

26 Big SQL V3.0: Bringing SQL on Hadoop to the next level Massively parallel SQL engine on Hadoop Architected from the ground up for low latency and high throughput Comprehensive SQL support The same SQL you use on your data warehouse should run with few or no modifications Full support for sub-queries All standard join operations Stored procedures / User defined functions Supports all modern file formats Big SQL Engine SQL MPP Run-time HDFS CSV Seq Parquet RC Avro ORC SQL Based Application JSON Custom Big Data and Real Time Analytics 26

27 Big SQL on POWER8 delivers Big Data queries faster than Hive on Ivy Bridge TCP-DS inspired workload queries BigInsights v3.0 on 8 S822L data nodes Time to complete queries in 7 concurrent streams 11.2x Faster S822L, 3.3 GHz 24 cores 256 GB Memory RHEL hours 4.7x 40 min Lower Cost CPO Master POWER Competitive Proof Points v2.9 27

28 Hadoop implementation pain points Building clusters is very complex, time consuming and can become costly Difficult to plan, architect and build out a cluster of nodes and network components Difficult to manage the cluster when deployed Inability to share resources results in cluster sprawl x86 deployments are not flexible enough Built with a fixed processor to disk ratio When you add more machines you get both processors and hard drives Some Hadoop workloads require more of one or the other but not always both at the same time HDFS NameNode is a single point of failure HDFS is not POSIX compliant and wastes storage Requires use of non standard commands and utilities Copies of data may need to be stored twice Once in Linux file system and once in HDFS Hadoop workload management is very basic Does not utilize cluster resources to their maximum Does not support multi-tenancy Can lead to silos of under utilized clusters Big Data and Real Time Analytics 28

29 Introducing the IBM Solution for Hadoop Power Systems Edition Best-in-class hardware IBM POWER Systems IBM InfoSphere BigInsights or Open-source Hadoop IBM DCS3700 Storage IBM Platform Symphony IBM Platform Cluster Manager Best-in-class software IBM BigInsights IBM Platform Computing IBM Platform Symphony IBM Platform Cluster Manager IBM Elastic Storage Formerly GPFS Distributed File System IBM Elastic Storage, HDFS Linux Operating Environment RHEL SUSE IBM Power Systems IBM Power 7+, Power8 Big Data and Real Time Analytics 29

30 Platform Symphony provides enterprise class Hadoop cluster management Replaces the open source MapReduce scheduler of Hadoop Existing Hadoop applications run unchanged Better performance and utilization in cluster overall and within individual jobs Scheduling is more efficient More efficient data transfer mechanisms for shuffle phase of MapReduce Uses generic slots rather than fixed map and reduce slots which waste resources Big Data and Real Time Analytics 30

31 Platform Symphony provides enterprise class Hadoop cluster management cont. Multi-tenancy is built in system is designed for concurrency Reduces costs through better resource sharing and utilization Provides dynamic priority based concurrency Interactive workloads like BigSheets can be given higher priority (.e.g., during the day) Priorities can be adjusted in real time Preemptive scheduling policies ensure service levels are met while also allowing the cluster to be fully utilized Jobs entering the system are immediately given the priority they are guaranteed Applications can decide if they allow other applications to use their unused capacity GUI provided to configure, manage and monitor policies and workload Allows running multiple versions of Hadoop Helps control costs and provides flexibility during upgrades Big Data and Real Time Analytics 31

32 Platform cluster manager removes the complexity of standing up clusters Provides comprehensive set of tools to install, provision and configure cluster management environment GUI, automated installation scripts and guidance Provides flexible provisioning options Bare-metal (i.e., no virtualization) KVM (virtualized cluster environment) Provides GUI for end users to provision, deploy, manage and monitor clusters Improves administrator and/or user productivity Deploy clusters easier and faster Saves on administration costs Lowers risk of deploying Hadoop clusters Big Data and Real Time Analytics 32

33 IBM Elastic Storage(GPFS) brings enterprise class file system capabilities to Hadoop No single point of failure like HDFS NameNode Metadata is distributed GPFS is POSIX compliant Any Linux command/utility can be run against data, not just Hadoop commands Easier to use and manage Makes archival and backup easier Other workload types can access data in cluster Saves storage space and administrator work Avoids moving data around to process inside and outside of HDFS Avoids having duplicate data inside and outside of HDFS Native OS Applications (POSIX) Allows variable block sizes to exist on same cluster to meet the needs of different applications Big Data and Real Time Analytics 33

34 IBM Solution for Hadoop Power Systems Edition key architecture components PowerLinux Data Node IBM PowerLinux 7R2 (P7+ based, P8 coming in 4Q2014) 2 sockets Power GHz CPU Data: 29 x 900Gb SAS HDDs, JBOD I/O Exp OS: 1 x 300Gb SAS HDD 128 GB DDR3 RDIMMs PowerLinux Management Node (JobTracker, NameNode, Console) IBM PowerLinux 7R2 (P7+ based, P8 coming in 4Q2014) 2 sockets Power GHz CPU OS: 6 x 600GB SAS HDD, mirrored 128GB DDR3 RDIMMs DCS3700 IBM DCS3700 Storage f/c c A maximum of 60 x 3.5" HDD, 4 TB each Standard controller (3000) validated 1GbE, 10GbE Switches 1GbE: IBM RackSwitch G GbE RJ45 ports, four 10 GbE SFP+ ports Low 130 W power rating and variable speed fans to reduce power consumption 10GbE: IBM RackSwitch G8264 Optimized for applications requiring high bandwidth and low latency Up to 64 1 Gb/10 Gb SFP+ ports, four 40 Gb QSFP+ports, 1.28 Tbps non-blocking throughput Big Data and Real Time Analytics 34

35 Learn more about Big Data Analytics Solutions on Power IBM Analytics Solutions for Power Systems The PowerLinux Community (DeveloperWorks) Big Data and Real Time Analytics 35