CS4491/CS 7265 BIG DATA ANALYTICS

Transcription

1 CS4491/CS 7265 BIG DATA ANALYTICS BIG DATA * The contents are adapted from Dr. Jeongkyu Lee@UB Mingon Kang, Ph.D Computer Science, Kennesaw State University

2 Era of Big Data 2011: Amount of Digital Information = 1.8 ZB 2020: maybe 50 times more?? Main Frame Computer Virtual Realty IoT Internet IT Mobile PC Mobile everywher Computer e www PC Broadband SNS Data Size Data Type Data Characteristic EB (Exa Byte) 90 = 100EB Structured Data (RDBMS, Office Info) Organized Data Beginning ZB 2011 = 1.8 ZB Unstructured Data (MM, SNS, ) Complex, SNS Data Big Data AI ZB Era 2020 = x 50 data Object, Spatial (IoT, RFID, Sensor) Real-time Data

3 What is BIG DATA? Wiki said in 2012 data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process the data within a tolerable elapsed time Wiki says NOW A broad term for data sets so large or complex that traditional data processing applications are inadequate. Challenges include analysis, capture, data curation, search,.. The term often refers simply to the use of predictive analytics or other certain advanced methods to extract value from data, and seldom to a particular size of data set.

4 What is BIG DATA? Wiki said in 2012 data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process the data within a tolerable elapsed time Wiki says NOW A broad term for data sets so large or complex that traditional data processing applications are inadequate. Challenges include analysis, capture, data curation, search,.. The term often refers simply to the use of predictive analytics or other certain advanced methods to extract value from data, and seldom to a particular size of data set.

5 What is BIG DATA? Gartner says Big data is high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization Oxford English Dictionary says big data n. Computing (also with capital initials) data of a very large size, typically to the extent that its manipulation and management present significant logistical challenges; (also) the branch of computing involving such data.

6 3V: Volume (Scale) Data Volume 44x increase from From 0.8 zettabytes to 35zb Data volume is increasing exponentially Exponential increase in collected/generated data

7 3V: Variety (Complexity) Various formats, types, and structures Text, numerical, images, audio, video, sequences, time series, social media data, multi-dim arrays, etc Static data vs. streaming data A single application can be generating/collecting many types of data To extract knowledge all these types of data need to linked together

8 3V: Velocity (Speed) Data is begin generated fast and need to be processed fast Online Data Analytics Late decisions missing opportunities Examples E-Promotions: Based on your current location, your purchase history, what you like send promotions right now for store next to you Healthcare monitoring: sensors monitoring your activities and body any abnormal measurements require immediate reaction

9 4V: Veracity

10 Who s Generating Big Data Mobile devices (tracking all objects all the time) Social media and networks (all of us are generating data) Scientific instruments (collecting all sorts of data) Sensor technology and networks (measuring all kinds of data) The progress and innovation is no longer hindered by the ability to collect data But, by the ability to manage, analyze, summarize, visualize, and discover knowledge from the collected data in a timely manner and in a scalable fashion

11 How to use big data

12 What s driving Big Data - Optimizations and predictive analytics - Complex statistical analysis and huge data mining - All types of data, and many sources - Very large datasets - More of a real-time - Ad-hoc querying and reporting - Basic data mining techniques - Structured data, typical sources - Small to mid-size datasets

13 How to use Big Data Big Data, like Business Intelligence, can be used to improve stuff. It can also be used to solve problems (i.e. answer big questions ).

14 How to use Big Data Suppose you have a Combine Harvester.

15 How to use Big Data Suppose you have a Combine Harvester. Sensors are becoming increasingly cheap, so it would be quite easy to cover the harvester in sensors (temperature, GPS, pressure, capacity, etc ). This will generate some big data. Especially if all the harvesters in Europe are equipped with the same sensors.

16 How to use Big Data But what would you use this data for? Finding the most economical driving style by monitoring driving habits, tracking the position of the harvester and fuel levels in the tank. Monitoring vibrations and temperature patterns in the parts to predict when parts might break. This could then tie into a system that automatically orders parts. Tracking the harvester s position and yield, to identify the most fertile areas and those which require fertilisers.

17 How to use Big Data

18 How to use Big Data How did Google use Big Data? They stored search history for every user as well as what every user clicked. This data was needless and pointless (Data Exhaust). What do you think that Google did with this data?

19 How to use Big Data Google used that data to power a spell-checker. Because if I search for bansnas and click on something relating to bananas, the chances are that I meant to search for bananas in the first place. About 2 billion searches a day are made on Google.

20

21 How to use Big Data LAPD use PredPol to predict crimes.

22 How to use Big Data The LAPD mined 13 million crime reports with a specialised algorithm. 1. Type of Crime 2. Place of Crime 3. Time of Crime 13 million arrests is 80 years of crime data. They then build mission maps covering dangerous areas, and would patrol them to minimise crime.

23 How to use Big Data It worked. It reduced: Property crime by 12% and Burglary by 26%

24 How to manage Big data

25 Challenges in Handling Big Data The Bottleneck is in technology New architecture, algorithms, techniques are needed Also in technical skills Experts in using the new technology and dealing with big data

26 Storing Big Data Here are a few tools that can be used to store Big Data.

27 Traditional Large-Scale Computation

28 Distributed System: Problems

29 Distributed Systems: Data Storage

30 Data-Driven World

31 Data Become the Bottleneck

32 Requirements for a new approach

33 Partial Failure Support

34 Data Recoverability

35 Component Recovery

36 Consistency

37 Scalability

38 Newbie for Big Data - Hadoop Eco-System 38

39 Hadoop History

40 Core Hadoop Concepts

41 Very High-level Overview

42 Fault Tolerance

43 CPSC651- Big Data Systems and Analytics 43

44 Hadoop in IBM

45 Hadoop in Oracle

46 Hadoop in Teradata

47 Hadoop in Microsoft

48 Hadoop in EMC

49