By: Shrikant Gawande (Cloudera Certified )

Size: px

Start display at page:

Download "By: Shrikant Gawande (Cloudera Certified )"

Sibyl Byrd
5 years ago
Views:

1 By: Shrikant Gawande (Cloudera Certified )

2 What is Big Data? For every 30 mins, a airline jet collects 10 terabytes of sensor data (flying time) NYSE generates about one terabyte of new trade data per day to Perform stock trading analytics to determine trends for optimal trades.

3 Facebook Example Facebook users spend 10.5 billion minutes (almost 20,000 years) online on the social network. Facebook has an average of 3.2 billion likes and comments are posted every day.

Twitter Example Twitter has over 500 million registered users. The USA, whose 141.8 million accounts represents 27.

4 Twitter Example Twitter has over 500 million registered users. The USA, whose million accounts represents 27.4 percent of all Twitter users, good enough to finish well ahead of Brazil, Japan, the UK and Indonesia. 79% of US Twitter users are more likely to recommend brands they follow. 67% of US Twitter users are more likely to buy from brands they follow. 57% of all companies that use social media for business use Twitter.

5 Hadoop is being used across industries Industries using Hadoop Source : Karmasphere

6 Why to learn Big Data?

7 What Big Companies Have To Say..

8 Data Volume Is Growing Exponentially Estimated Global Data Volume: 2011: 1.8 ZB 2015: 7.9 ZB The world's information doubles every two years Over the next 10 years: The number of servers worldwide will grow by 10x Amount of information managed by enterprise data centers will grow by 50x Number of files enterprise data center handle will grow by 75x Source: was based on the 2011 IDC Digital Universe Study

IBM s Definition IBM s definition Big Data Characteristics http://www-1.ibm.

9 IBM s Definition IBM s definition Big Data Characteristics A collection of large and complex data sets which are difficult to process using common database management tools or traditional data processing applications. Big Data is the amount of data that is beyond the storage and the processing capabilities of a single physical machine. Data that has extra large volume, comes from variety of sources, variety of formats and comes at us with a great velocity it normally referred as Big Data

10 It s more of unstructured Data than Structured Data

11 A Traditional Approach Under Pressure

12 Why Big Data? ERP CRM Data ( few TBs) Enterprise data What Data We have been adding in last 3-4 Years Customer Experience Click Streams Online Campaign Banner Ads capturing every click 100 n TBs User Entered data Search In product search Social media to understand general sentiments

Common Business Applications Industry Use Cases Types of Data New Account Risk Screens Text, Server Logs Financial Services Trading Risk Server Logs Insurance Underwriting Geographic, Sensor, Text

13 Common Business Applications Industry Use Cases Types of Data New Account Risk Screens Text, Server Logs Financial Services Trading Risk Server Logs Insurance Underwriting Geographic, Sensor, Text Call Details records (CDR) Machine, Geographic Telecom Infrastructure Investment Machine, Server Logs Real-Time Bandwidth Allocation Server Logs, Text, Social 360 Degree View of Customer ClickStream, Text Retail Localized, Personal Promotion Geographic Website Optimization ClickStream Supply Chain and Logistics Sensor Manufacturing Assembly Line Quality Assurance Sensor Crowd sourced Quality Assurance Social HealthCare Use Genomic in Medical Trials Structured Monitor Patient Vitals in Real-Time Sensor Pharmaceuticals Recruit and Retain Patients for Drug Trails Improve Prescription Adherence Social, Clickstream Social, Unstructured, Geographic Oil and Gas Unify Exploration and Production Data Monitor Rig Safety in Real Time Sensor, Unstructured, Geographic Sensor, Unstructured

14 How can we find products that customers are interested in BUT DON T BUY?

15 Leveraging ALL Business Data How to Extract Insights from 9TBs of Web Logs? How do you make sense of this?

16 Leveraging ALL Business Data How to Extract Insights from 9TBs of Web Logs? What users did when they come to our web site? Which product they viewed? Which product seen but not purchased? Why? New Offering based on past data? In the First line User has seen some product by some particular ID?

17 Leveraging ALL Business Data How to Extract Insights from 9TBs of Web Logs? (Contd Visitor views 2nd product - We want to do this not just for 1 customer but all the customers

18 Hidden Treasure Insight into data can provide Business Advantage. Some key early indicators can mean Fortunes to Business. More Precise Analysis with more data New offerings to the customer

19 Limitations of Existing Data Analytics Architecture

20 Solution: A Combined Storage Computer Layer

21 Differentiating factors

22 Some of the Hadoop Users

23 Why DFS?

24 What is Hadoop? Apache Hadoop is a framework that allows for the distributed processing of large data sets across clusters of commodity computers using a simple programming model. It is an Open-source Data Management with scale-out storage & distributed processing

25 Hadoop Key Characteristics

26 Hadoop History

27 Hadoop Eco-System

28 Hadoop Core Components HDFS Hadoop Distributed File System(Storage) Distributed across nodes Natively redundant Name Node tracks locations. MapReduce (Processing) Splits a task across processors near the data & assembles results Self-Healing, High Bandwidth Clustered storage

29 Hadoop Core Components (contd.)

30 HDFS Architecture

DataNodes slaves which are deployed on each machine and provide

31 Main Components of HDFS NameNode master of the system maintains and manages the blocks which are present on the DataNodes DataNodes slaves which are deployed on each machine and provide the actual storage responsible for serving read and write requests for the clients

32 NameNode and Datanode

33 NameNode Meta Data Meta-data in Memory The entire metadata is in main memory No demand paging of FS meta-data Types of Metadata List of files List of Blocks for each file List of DataNode for each block File attributes, e.g. access time, replication factor A Transaction Log Records file creations, file deletions. etc

34 Storage : Name-Node and Data-Node.S Processing : Job-Tracker and Task-Tracker.S H1 H2 H3 H4

35 Poll - 01

36 Poll - 02

37 Poll - 03

38 Poll - 04

39 Poll - 05

41 Hadoop Courses and its fees across major training institutes

42 Hadoop Course fee at Cloudera Cloudera Hadoop Training :

43 Hadoop Course fee at HortonWorks and Edureka $ 2,795 = Rs. 1,73,290

44 My Contact Details:

45 Thank You

Table of Contents. Are You Ready for Digital Transformation? page 04. Take Advantage of This Big Data Opportunity with Cisco and Hortonworks page 06

Table of Contents. Are You Ready for Digital Transformation? page 04. Take Advantage of This Big Data Opportunity with Cisco and Hortonworks page 06 Table of Contents 01 02 Are You Ready for Digital Transformation? page 04 Take Advantage of This Big Data Opportunity with Cisco and Hortonworks page 06 03 Get Open Access to Your Data and Help Ensure