IBM BigInsights - Hadoop jako rozwiązanie korporacyjne. Tomasz Zawadzki Dyrektor Zarządzający Atom-tech

Size: px

Start display at page:

Download "IBM BigInsights - Hadoop jako rozwiązanie korporacyjne. Tomasz Zawadzki Dyrektor Zarządzający Atom-tech"

Walter Leonard
5 years ago
Views:

1 IBM BigInsights - Hadoop jako rozwiązanie korporacyjne Tomasz Zawadzki Dyrektor Zarządzający Atom-tech

2 IBM BigInsights - Hadoop jako rozwiązanie korporacyjne Tomasz Zawadzki Dyrektor Zarządzający Atom-tech

Information is at the Center of a New Wave of Opportunity 44x as much Data and

Insights 1 in 3 Business leaders frequently make decisions based on information

information 1 in 2 they need to do their jobs 2009 800,000 petabytes 80% Of world

part of their visionary plans to enhance competitiveness 60% of CEOs need to do a

3 Information is at the Center of a New Wave of Opportunity 44x as much Data and Content Over Coming Decade zettabytes And Organizations Need Deeper Insights 1 in 3 Business leaders frequently make decisions based on information they don t trust, or don t have Business leaders say they don t have access to the information 1 in 2 they need to do their jobs ,000 petabytes 80% Of world s data is unstructured 83% of CIOs cited Business intelligence and analytics as part of their visionary plans to enhance competitiveness 60% of CEOs need to do a better job capturing and understanding information rapidly in order to make swift business decisions

Big Data Presents Big Opportunities Extract insight from a high volume, variety and velocity of data in a timely and cost-effective manner Variety: Velocity:

4 Big Data Presents Big Opportunities Extract insight from a high volume, variety and velocity of data in a timely and cost-effective manner Variety: Velocity: Volume: Manage and benefit from diverse data types and data structures Analyze streaming data and large volumes of persistent data Scale from terabytes to zettabytes

Customer Engagements Common use cases Analyze machine data (logs, sensors, etc.) to reduce operational risk, predict / identify / correct problems, improve customer service, etc.

5 Customer Engagements Common use cases Analyze machine data (logs, sensors, etc.) to reduce operational risk, predict / identify / correct problems, improve customer service, etc. Analyze social media data and customer behavior for to improve customer retention, brand management, lead generation, operations, etc.... Text, Blog, Weblog Click streams Log & transactions Technology mandates Extract business insight from large volumes of raw data (often outside operational systems) Integrate with existing software Ready for enterprise use Consumer Insight Multi-channel sales Next Gen Fraud Models Text Analytics Biological Sequences Operational system & streams data sources New Business Development Statistical Model Building

IBM Big Data Platform Strategy Integrate and manage the full variety, velocity and volume

available data for ad-hoc analysis Development environment for building new analytic

governance Integrate with enterprise software Analytic Applications BI / Reporting

Hadoop System Accelerators Industry App Application Development Stream Computing

6 IBM Big Data Platform Strategy Integrate and manage the full variety, velocity and volume of Big Data Apply advanced analytics to information in its native form Visualize all available data for ad-hoc analysis Development environment for building new analytic applications Support workload optimization and scheduling Provide for security and governance Integrate with enterprise software Analytic Applications BI / Reporting Exploration / Visualization Functional App IBM Big Data Platform Visualization & Discovery Hadoop System Accelerators Industry App Application Development Stream Computing Predictive Analytics Information Integration & Governance BI Content / Reporting Analytics Systems Management Data Warehouse

Enterprise class Platform for volume, variety, velocity Enhanced Hadoop foundation Analytics Text analytics & tooling Application accelerators Usability Web console

BigInsights Apache Hadoop Basic Edition Free download Integrated install Online InfoCenter BigData Univ.

7 Enterprise class Platform for volume, variety, velocity Enhanced Hadoop foundation Analytics Text analytics & tooling Application accelerators Usability Web console Spreadsheet-style tool Ready-made apps Enterprise Class Storage, security, cluster management Integration Connectivity to Netezza, DB2, JDBC databases, etc InfoSphere BigInsights Apache Hadoop Basic Edition Free download Integrated install Online InfoCenter BigData Univ. Enterprise Edition Licensed Application accelerators Pre-built applications Text analytics Spreadsheet-style tool RDBMS, warehouse connectivity Administrative tools, security Eclipse development tools Performance enhancements.... Breadth of capabilities

8 Function BigInsights Content Version Basic Edition Integrated Install Inc Inc Hadoop (including common utilities, HDFS, MapReduce framework) Inc Inc Jaql (programming / query language) Inc Inc Pig (programming / query language) Inc Inc Flume (data collection/aggregation) Inc Inc Hive (data summarization/querying) Inc Inc Lucene (text search)* Inc Inc Zookeeper (process coordination) Inc Inc Avro (data serialization) Inc Inc HBase (real time read/write) Inc Inc HCatalog (table and storage management service) Inc Inc Sqoop (RDBMS bulk data transfer) Inc Inc Oozie (workflow/ job orchestration) Inc Inc Online documentation Inc Inc Integration with JDBC sources through general-purpose Jaql module Inc Inc Integration with DB2 (sample functions to submit jobs, read data) Inc Inc Enterprise Edition

9 Function BigInsights Content (cont d) Basic Edition Integration with R (Jaql module to invoke R statistical capabilities from BigInsights) n/a Inc Integration with Netezza, DB2 LUW with DPF from Jaql n/a Inc LDAP authentication, Guardium support, etc. n/a Inc Integrated Web Console n/a Inc Business process accelerators (social data, machine data analytics) n/a Inc Platform performance enhancements (Adaptive MapReduce, large scale indexing, efficient processing of compressed text files, flexible job scheduler, etc.) Text analytics n/a Inc Eclipse tools for text analytic development, Jaql, Hive, Java n/a Inc Applications for data import/export, Web crawl, machine learning, etc. n/a Inc Web-based application catalog n/a Inc Spreadsheet-like analytical tool n/a Inc IBM support Opt Inc Streams, Data Explorer, Cognos BI (limited use licenses) n/a Inc Unlimited storage n/a Inc n/a Enterprise Edition Inc

10 Support for other Hadoop Distributions Installation option for BigInsights Enterprise Edition Support Hadoop distributions from third parties, beginning with Cloudera Configures key BigInsights technologies to work with Cloudera foundation Easy way for Cloudera customers to leverage leading BigInsights capabilities Text analytics BigSheets (spreadsheet-style tool for business analysts) Eclipse plug-ins for rapid application development Ready-made sample applications for Web crawl, social media data collection, data import/export, ad hoc query, etc. Administrative tooling

BigInsights: Value Beyond Open Source Open source components Enterprise Capabilities Visualization & Exploration Development Tools Advanced Engines Connectors Workload Optimization Administration &

integration Spreadsheet-style analysis Integrated installation of supported open source and other components Web Console for admin and application access Platform enrichment: additional security,

11 BigInsights: Value Beyond Open Source Open source components Enterprise Capabilities Visualization & Exploration Development Tools Advanced Engines Connectors Workload Optimization Administration & Security IBM-certified Apache Hadoop or or Key differentiators Built-in analytics Text engine, annotators, Eclipse tooling Interface to project R (statistical platform) Enterprise software integration Spreadsheet-style analysis Integrated installation of supported open source and other components Web Console for admin and application access Platform enrichment: additional security, performance features,... World-class support Full open source compatibility Business benefits Quicker time-to-value due to IBM technology and support Reduced operational risk Enhanced business knowledge with flexible analytical platform Leverages and complements existing software

12 BigInsights and the data warehouse Big Data analytic applications Traditional analytic tools Data warehouse BigInsights Filter Transform Aggregate

13 BigInsights and the data warehouse Traditional analytic tools Big Data analytic applications BigInsights Data Warehouse Query-ready archive for cold warehouse data

14 About the BigInsights Platform Flexible, enterprise-class support for processing large volumes of data Based on Google s MapReduce technology Inspired by Apache Hadoop; compatible with its ecosystem and distribution Well-suited to batch-oriented, read-intensive applications Supports wide variety of data Enables applications to work with thousands of nodes and petabytes of data in a highly parallel, cost effective manner CPU + disks = node Nodes can be combined into clusters New nodes can be added as needed without changing Data formats How data is loaded How jobs are written

15 "Map" step: Input split into pieces The MapReduce Programming Model Worker nodes process individual pieces in parallel (under global control of the Job Tracker node) Each worker node stores its result in its local file system where a reducer is able to access it "Reduce" step: Data is aggregated ( reduced from the map steps) by worker nodes (under control of the Job Tracker) Multiple reduce tasks can parallelize the aggregation

16 Logical MapReduce Example: Word Count map(string key, String value): // key: document name // value: document contents for each word w in value: EmitIntermediate(w, "1"); reduce(string key, Iterator values): // key: a word // values: a list of counts int result = 0; for each v in values: result += ParseInt(v); Emit(AsString(result)); Content of Input Documents Hello World Bye World Hello IBM Map 1 emits: < Hello, 1> < World, 1> < Bye, 1> < World, 1> Map 2 emits: < Hello, 1> < IBM, 1> Reduce (final output): < Bye, 1> < IBM, 1> < Hello, 2> < World, 2>

17 Input Documents MapReduce Processing Hello World Bye World Hello IBM Map 1 emits: < Hello, 1> < World, 1> < Bye, 1> < World, 1> Map 2 emits: < Hello, 1> < IBM, 1> Reduce (final output): < Bye, 1> < IBM, 1> < Hello, 2> < World, 2>

18 So What Does This Result In? Easy To Scale Fault Tolerant and Self-Healing Data Agnostic Extremely Flexible

19 Seamless process for single node and cluster environments Integrated installation of all selected components Post-install validation of IBM and open source components Get up and running quickly! Web Installation Tool No need to iteratively download, configure, and test multiple open source projects and pre-requisite software.

Manage BigInsights Inspect /monitor system health Add / drop

(applications) Explore / modify file system Create custom

.. Launch applications Spreadsheet-like analysis tool

20 Manage BigInsights Inspect /monitor system health Add / drop nodes Start / stop services Run / monitor jobs (applications) Explore / modify file system Create custom dashboards... Launch applications Spreadsheet-like analysis tool Pre-built applications (IBM supplied or user developed) Publish applications Monitor cluster, applications, data, etc. Web Console

21 Web-based analysis and visualization Spreadsheet-style Analysis Spreadsheet-like interface Define and manage long running data collection jobs Analyze content of the text on the pages that have been retrieved

Big Data Application Ecosystem Data integration scenario: Pre-defined work flows simplify loading data from various sources Work flows can be configured, deployed, executed and scheduled Application

22 Big Data Application Ecosystem Data integration scenario: Pre-defined work flows simplify loading data from various sources Work flows can be configured, deployed, executed and scheduled Application scenarios (web log, , social media, ): Samples provide starting point, speed time to value Big Data Web Console Publish Eclipse Development tooling: Text analytics MapReduce Query languages... App Development App library MapReduce, Text Analytics Query App Development Code application program, and generate associated App Deploy Apps to Enterprise Manager

Quick start sample applications 20+ software assets based on common customer needs Useful for starting point for various applications Accessible through Web console Available assets Data movement

23 Quick start sample applications 20+ software assets based on common customer needs Useful for starting point for various applications Accessible through Web console Available assets Data movement From relational DBMS, files, REST-based sources To relational DBMS, files Web crawler, social media data collectors, etc. Ad hoc queries for Jaql, Hive, Pig Machine learning algorithms and data processing Data sampling and subsetting TeraGen-TeraSort, WordCount sample applications

24 Running Applications from the Web Console Import & Export Data Database & Files Web and Social Analyze and Query Predictive Analytics Text Analytics SQL/Hive, Jaql, Pig, HBase

25 Build a Big Data Program MapReduce example Eclipse tools For Jaql, Hive, Pig Java MapReduce, BigSheets plug-ins, text analytics, etc.

26 Quickly drag and drop to create new applications

... Add, group, or remove widgets for: BigSheets collections and charts Cluster/system Monitoring

27 Visualize results through dashboards Built-in dashboards for monitoring system health, application status, distributed file system, etc. Easy to customize.... Add, group, or remove widgets for: BigSheets collections and charts Cluster/system Monitoring HDFS monitoring MapReduce metrics Third party Widgets or Open Social Gadgets can be added to a dashboard Create new, custom dashboards to suit your needs!

BigInsights and Text Analytics Distills structured info from unstructured text Sentiment analysis Consumer behavior Illegal or suspicious activities Parses text and detects meaning with

Early in the second half, Netherlands striker, Arjen Robben, had a breakaway, but the keeper for Spain, Iker Casillas made the save. Winger Andres Iniesta scored for Spain for the win.

28 BigInsights and Text Analytics Distills structured info from unstructured text Sentiment analysis Consumer behavior Illegal or suspicious activities Parses text and detects meaning with annotators Unstructured text (document, , etc) Football World Cup 2010, one team distinguished themselves well, losing to the eventual champions 1-0 in the Final. Early in the second half, Netherlands striker, Arjen Robben, had a breakaway, but the keeper for Spain, Iker Casillas made the save. Winger Andres Iniesta scored for Spain for the win. Understands the context in which the text is analyzed Features pre-built extractors for names, addresses, phone numbers, etc. Classification and Insight Built-in support for English, Spanish, French, German, Portuguese, Dutch, Japanese, Chinese

29 BigInsights Text Analytics Development

30 Example Analysis : Extraction from Twitter messages Extract intent, interests, life events and micro segmentation attributes Monetizable Intent Relocation Name, Birth Day Location I had an iphone, but it's (I've no idea where it's)!want a blackberry im moving to miami in 3 months. i look foward to the new good!!! U shouldnt! Think about the important stuff, like ur birthday ;) btw happy birthday Sylvia ;) I'm at Mickey's Irish Pub Downtown (206 3rd St, Court Ave, Des Moines) w/ 2 others While accounting for less relevant messages Subtle Spam, Advertising I think deserves his 2 AMAZING songs in top ten!!! Buy them on itunes Looking to buy a phone? WiFi Cell Phones, Windows Mobile Sarcasm, Wishful Gotta do more research my Versace term paper 2day. Before I die, I want a versace purple diamond tiara. Im just sayin>lol had so much fun today! I want to buy a million dollar house with a wrap around porch wading river on the long island sound, ha i wish!

31 Unified tooling environment for the data analytics lifecycle Sample data (if desired) Develop, test, and deploy analytic applications 1. Sample your Data 2. Develop your application using BigInsights tools Administer, execute, and monitor the deployed applications 5. Deploy your application on the cluster 3. Test your application 4. Package and publish your application

Application Accelerators Quickly build, deploy custom applications in high-value areas IBM Accelerator for Social Data

utility, healthcare, travel and transportation, CPG, Retail, etc.

proactive maintenance, troubleshooting, outage prevention, efficiency tracking, etc Ships with BigInsights v2 IBM

32 Application Accelerators Quickly build, deploy custom applications in high-value areas IBM Accelerator for Social Data Analytics B2C businesses Sample applications: Customer acquisition / retention, Customer Segmentation or Micro Segmentation, Marketing Campaign Optimization, Lead generation, Brand Management or Surveillance Ships with BigInsights v2 and Streams v3 IBM Accelerator for Machine Data Analytics Cross-industry: manufacturing, oil & gas, energy and utility, healthcare, travel and transportation, CPG, Retail, etc. Operational efficiency monitoring, security incident investigation. proactive maintenance, troubleshooting, outage prevention, efficiency tracking, etc Ships with BigInsights v2 IBM Accelerator for Telco Event Data Analytics Telcos Campaign management, real-time promotion, fraud detection, service assurance and network monitoring, Ships with Streams v3, but works with BigInsights or PureSparta for Analytics (a.k.a. Netezza)

Flexible job scheduler option Performance enhancements Optimize response time for small jobs Available in addition to FAIR, FIFO scheduling

33 Flexible job scheduler option Performance enhancements Optimize response time for small jobs Available in addition to FAIR, FIFO scheduling Adaptive MapReduce Speeds up a class of jobs (e.g., jobs that process small files) Accomplished by changing how certain MapReduce tasks executed Mappers can decide at runtime to take on more work (until it doesn t make sense anymore). Communication via ZooKeeper. Enabled through Jaql option, MapReduce job property setting Efficient processing of compressed text data Use multiple Map tasks (vs Hadoop default of 1) for processing compressed text files Enabled through BigInsights LZO-based compression technology Automatic with Jaql; programming option with Java MapReduce

34 BigInsights Connectivity to DBMS / Warehouse DB2 LUW, IW with DPF Netezza BigInsights BigInsights drives RDBMS work DBMS drives BigInsights work JDBC DBMS

35 On Big Data, BigInsights and Atom-tech... Big Data is our strategic initiative as an application of cloud Broad experience in cloud & infrastructure projects Vendor-independent hypervisor vision of IT IBM InfoSphere BigInsights & Atom-tech Tool to exploit growing variety, velocity, and volume of data Delivers diverse range of analytics to our customers Leverages and extends open source aligned with our vision of business Provides enterprise-class features and supporting services we like it Complement existing software investments and commercial offerings Available in basic (free) and enterprise editions we can easily provide POCs & POTs Atom-tech has a strong partnership with IBM to deliver efficient solutions

36 Pytania? Tomasz Zawadzki Dyrektor Zarządzający Atom-tech

37 Dziękuję! Tomasz Zawadzki Dyrektor Zarządzający Atom-tech

IBM Big Data Summit 2012

IBM Big Data Summit 2012 12.10.2012 InfoSphere BigInsights Introduction Wilfried Hoge Leading Technical Sales Professional hoge@de.ibm.com twitter.com/wilfriedhoge 12.10.1012 IBM Big Data Strategy: Move