TEQIP Short Term course on Big Data 7-11 Aug 2017

Transcription

1 TEQIP Short Term course on Big Data 7-11 Aug 2017 Veena Bansal Workshop Coordinator Associate Professor, IIT Kanpur Visiting Associate Professor, IIT Bhilai

2 Big data A misnomer Big refers to the volume of data Only one aspect of data Datayard Datalake

3 Big Data Era Data Torrent + Cloud computing (computing anywhere, anytime) Big Data Era

4 Big data The 3 V s: Volume, Variety, Velocity Too big to fit into memory of one server What is the size of the memory of a server? RAM (not hard disk) Video captured daily > first 50 years of TV Facebook photographic data > all pixels processed by Kodak

5 Facts and Figures Walmart handles 1 million customer transactions/ hour. Facebook handles 40 billion photos from its user base! Facebook inserts 500 terabytes of new data every day. A flight generates 240 terabytes of flight data in 6-8 hours of flight.

6 An Insight Byte : One grain of rice KB (3): One cup of rice: MB (6): 8 bags of rice: GB (9): 3 Semi trucks of rice: TB (12): 2 container ships of rice PB (15): Blankets ½ of Jaipur Exabyte (18): Blankets West coast Or 1/4 th of India Zettabyte (21): Fills Pacific Ocean Yottabyte: An earth-sized rice bowl Desktop Internet Big Data Future

7 Big data The 3 V s: Volume, Variety, Velocity Too unstructured to fit into row-column structure of an RDBMS Structured data (give example: draw a table) Unstructured- twitter/facebook Semi-structured: web data Most difficult aspect of big data

8 Big data The 3 V s: Volume, Variety, Velocity Data generated by Machines Real time sensors and industrial machinery or vehicles. Web logs, personal health trackers etc. People Organizations

9 Big data The 3 V s: Volume, Variety, Velocity Data generated by Machines People With human generated data, we really refer to the vast amount of social media data, status updates, tweets, photos and videos. Organizations

10 Big data The 3 V s: Volume, Variety, Velocity Data generated by Machines People Organizations traditional types of data including transaction information data bases and structure data often stored in data warehouses.

11 Big data The 3 V s: Volume, Variety, Velocity Too continuously flowing to fit into a static data warehouse More than historical data

12 Big Data: Deconstructing the term Big data is characterized by: Lack of structure Diverse sources New data types Streaming data

13 Big data: Objective To convert the VVV data into new products/services: create value.5% data is analyzed Available: 2.8 ZB (Z is 10^21)

14 Facts and Figures The V s of Big data! Volume Velocity Variety Veracity Value Velocity Volume Veracity, Value Variety

15 Connectedness Size Volume Valence Value Velocity Variety Veracity Speed Complexity Quality

16 Veracity Veracity refers to the biases, noise, and abnormality in data, or the unmeasurable certainty in the truthfulness and trustworthiness of data

17 Valence Valence refers to the connectedness of big data. Such as in the form of graph networks.

18 Value Integrating data Reduce data complexity increase data availability unify your data systems all 3 above will lead to increased data collaboration -> add value to your big data

19 Value Analyzing data at scale It is important to pick right tools for scalability Big data + Analysis Question -> insight It is an iterative process Deep learning

20 Status In 2012: 20% organization: Using data from diverse sources 22% Working with new data types 12% streaming data 41% volume > 1 TeraByte In 2013: 28% using big data 23% have strategy 6% considering 3.5% making effective use of big data

21 Big data and traditional analytics Type of data Volume of data Analysis method Primary purpose Big Data Unstructured 100 terabytes to petabytes Machine Learning Data-based products Traditional data Formatted in rows and columns Tens of terabytes or less Hypothesis based Internal decision support and services

22 Big data: Transactions + Interactions + Observations Peta Byte Tera Byte Giga Byte CRM Customer Data Customer Segmentation Customer Support Customer Contact WEB BIG DATA Mega Byte ERP Purchase data Sales data Payment Data Inceasing data variety and complexity

23 Big data: Transactions + Interactions + Observations Peta Byte Tera Byte Giga Byte Meg a Byte BIG DATA Sensors(IoT)/ RFID/Devices User click system WEB Web logs Dynamic Pricing Search Marketing Behavioral targeting CRM ERP Mobile Web HD Video/audio/images Sentiments User/App/System generated content Business DBs Product logs SMS/MMS External Demographics Social Interactions and Feeds

24 Big data is not data driven decision making

25 McFarlan s Matrix Paradigm shift Gain Reengineering Automation rationalization Risk

26 Initial Applications of big data Exploratory in nature

27 Who are you Smart Ambitious Somehow associated with business HR/Marketing/Supply Chain/IT Online companies - facebook, google, linkedin are all data oriented

28 Why should you bother to know Big data is important for every company What aspect of big data to focus on Big data involves new technology/ architecture What skill set is required- learning/hiring You want to make most effective use of big data

29 Big Data: Possibilities

30 Challenge We regards content as cost but content provides an opportunity. How? Facebook: Ads Amazon: Recommendations Google: Anything that sells! Requires mindset change

31 Big data industries: Pure play Google ebay LinkedIN Facebook: built around big data from the beginning.

32 Big data industries Industries that generate lots of data and may benefit from big data: 1. Retail 2. Travel and Transportation 3. Telecommunication 4. Media and Entertainment 5. Financial Services Two ways to benefit: better decisions new business opportunities

33 Big Data Possibilities: pick one or more from each column to define your scenario Style of Data Sources of Data Industry Function Large Volume Online Financial Services Marketing Unstructured Video Healthcare Supply Chain Continuous Flow Multiple formats Sensor Manufacturing Human Resource Genomic Travel/ Finance Transport

34 Possibilities(contd..) You could plan to work with video data from your ATMS and branches to better understand customer behaviour/relationships OR Plan to combine electronic medical records and genomic data to create personalized treatment regimen for patients.

35 Possibilities(contd..) Data doesn t have to be big - use it to your advantage! If it works, call it Big Data Project (BDP) OR Massive Data Project (MDP) or whatever earns you credit/market/fund

36 Is Big Data here to stay? Improve decision about your business from your data is an old idea: UPS (1954) set up an analytics group. Data generated daily: 2.5 X 10^18 bytes What primarily generates this data? IoT (Fifty billion sensors) Social Media This data can be managed to optimize usage, consumption and provision of services.

37 Terminology Term Time Frame Specific Meaning Decision support Executive Support OLAP(Online Analytical Processing) Business Intelligence Use of data analysis to support decision making Focus on data analysis for decisions by executives Software for analyzing multidimensional data tables Tools to support data-driven decisions, with focus on reporting.

38 Terminology Term Time Frame Specific Meaning Analytics Big Data present Focus on Statistical and Mathematical analysis for decisions. Focus on very large, unstructured, fastmoving data.

39 Big data impact Sensors inside an organ to monitor health of the organ Sensors in the shoes/cars/bicycles to monitor speed Sensors used by athletes to track their regular activities, plan new routes and share their experience with others. Sensors for sleep analytics, mood analytics, fitness analytics.

40 Big data impact All these sensors generate enormous amount of data every hour Companies like IBM,SAP, Microsoft, Oracle, EMC, HP investing immensely in big data -> Big data is here to stay.

41 What s new from Management Perspective Big data groups operational in verticals like: Marketing Finance Product Management Strategy IT Data Analytics more focused on internal decision making Big data more focused on Customer facing products and services.

42 Examples LinkedIN uses Big Data for: People you may know Groups you may know Jobs you may be interested in Who has viewed your profile GE : optimizing the service contracts of industrial products.

43 Examples Google: Ad serving algorithms Zynga: Game serving algorithm Netflix: Improving movie recommendations Kaplan: Effective learning and test preparation strategies.

44 Contd.. Data Analytics: taking a pool of data, model, train, test and use for generating advice for decision making. Big data: on-going fast flowing stream of data. Continuous approach to sampling, analysis, acting on data required. Eg: Sentiment Analysis: assessing whether comments made about a company s brands and products are positive or negative or neutral.

45 Contd.. Potential problem for managers: viewing stream of analysis and reports without taking any action.eg: Sentiment is up, no it s down,.. Hooray, it s up again UN s Global Pulse Innovation Lab has developed HunchWorks (a big data related tool) which is monitoringoriented.

46 Turning Big Data into value

47 The structure of big data

48 The structure of big data Structured Most traditional data sources Unstructured Video data, audio data

49 New Management Orientation from Big Data For long, the focus has been on stability of automating processes related to marketing, sales and services. Big data flips this approach on its head-everything around is changing and the organizations who reckons this change the fastest has the upper hand. The prized business and IT capabilities are discovery and agility rather than stability.

50 New Management Orientation from Big Data This change can be incorporated swiftly in small startups but requires a sea of change in large organizations

51 Contd.. The military is accumulating data faster than it can analyze. There aren t enough human analysts to analyze the huge volumes of videos captured by the drones. The Air force is working on methods to analyze these data with less human intervention. Spurred on by the fact that if the analysis is secret, there s always a tiny probability of leak.

52 Contd.. In big data, you are looking for sandbox architecture that will facilitate/have functionalities to Integrate new data Handle ad hoc queries Visualize to accelerate human understanding.

53 Need of the hour Adopt new methodologies for insight and data-based products. Traditional waterfall methods (highly structural methods that yield a result at the end of a long process) don t work. Need to adopt agile processes: create small deliverables fast. Big data is often external to an organization using it.

54 Need of the hour Internal data of an organization can be profitably supplemented with external data. In a nutshell, the needs are: New technologies/algorithms/hardware Tools Skill set Mindset Cross fertilization of data/skills and domain from inside and outside of an organization.

55 New Opportunities (Strategy) from Big Data Value classes for an organization: Cost-reduction. Decision improvement Improvements in new products/services

56 New Opportunities from Big Data There are primarily 3 approaches to get excellent prediction using big data analytics: New/more data. More sophisticated algorithms Both of the above There is essentially a fourth approach: Multiple algorithms helping one another => Cognitive Computing => Watson.

57 Contd.. If you have data pertaining to customer attrition, you could improve prediction accuracy by adding data from these customers transaction histories. If you have a model for generating the next best offer, you could improve it by analyzing some of the customers comments and likes on social media sites. Some of these data may be big or some may not be, the key is to broadly look for new data sources to improve your models.

58 More examples Whole of Quora is based on big data. LinkedIN: People you may know (PYMK) feature recommending people you may connect with. Mostly everything on Facebook and Twitter. GE: sensor data for all devices that have a spinning turbine. Generally speaking, if efficiency is improved by a mere percentage point, it will save the host company millions of dollars.

59 Industries well suited to Big Data That have lot of data That were/are doing some kind of analysis Historical industry use of data and analytics: Data disadvantaged Health-care organizations Underachievers Traditional bank Overachievers Consumer products B2B firms Telecom Insurance Industrial Media & Online products Entertainment Retail Travel & Transport Electrical utilities Credit Cards

60 Data disadvantaged o Healthcare: Data mostly unstructured o B2B: Few Customers Little Data o Industrial Product Manufacturers: Few Customers Little Data o B2B2C: Retailers come in between Didn t know their customers

61 Underachievers: Lots of data but not used effectively o Telecom Firms: Had data but due to monopoly didn t feel challenged. o Media & Content: Continue to make decisions based on intuition. No mechanism to figure out if people were actually watching their content o Retailer: Had lots of customer data but didn t make use of it.

62 Underachievers: Lots of data but not used effectively o Banks: Had massive data but made very little use of it. o Electricity: Electricity cannot be stored; production must match consumption in real time

63 What can be done Healthcare: o Storing and analyzing text data using NLP. o CAT/MRI/Ultrasound images. o Genome data o Connected health: Tele-medicine and remote monitoring o Quantified self: Personal monitoring The challenge in healthcare: not how to gather data, but how to make use of it all!

64 Contd.. B2B o Service Records to improve products. o Customer Relationship Management B2B2C o P&G: Online sales not only to increase sales, but to understand customers better o Walmart: To share and analyze large volumes of retail point-of-scale data. o Smart medicine cabinet: Send reminder to the patient.

65 Contd.. Telecom: o Different prices. o Churning o Influential customers who can motivate others.

66 Contd.. Media: o Predict success of a show/movie before it is produced. o Discover taste of viewers Banks: o Know customers financial needs. o Know the products to recommend to fulfill those needs.

67 Big Data Key Business Functions: Marketing: o Data sources: Customers social data Customers media data Customers media data o Omni-channel relationship Customers move seamlessly between multiple physical and virtual touch-points. What and when customers say on social media What and when a customer walks into a store

68 Contd.. o Calls for customer help: Integrate and analyze data from multiple channels-structured/unstructured/small/big o Accurately attribute sales to variety of ads o Prioritize/ allocate marketing budget accordingly. o Prepare for situations when a new marketing or communication channel is created

69 Contd.. Sales: o Who is spending time how o Conversion rates vs. conversion prediction o Mobile phone location/ car location Supply Chain: o RFID as a means of monitoring supply chain movements. o GPS tracking on trucks and trains to optomize. o Supply chain is a complex system: Bring in details from e-commerce.

70 Contd.. o RFID and telematics sensors primarily tracks locations but ILC ( identification, location and condition) sensors can monitor the conditions of goods as well as parameters like: Light Temperature Tilt Angle Gravitational (G)-forces Whether a package has been opened

71 Contd.. Manufacturing: o Mass manufacturing/ Made to stock. o Made to order. o Discrete Manufacturing. o Optimize maintenance/ Minimize breakdown inventory optimization. HR: o Use employee locational data to analyze Employee interaction Group Collaborative groups Communication

72 What we don t know Impact on organizational structure One thing for sure: Big data projects will not be done in silos, it will be an organizational level project. Impact on Customer Relationship Will it lead to much more targeted behavior from the company towards the customers? Will the customers like it? Will there be a backlash?

73 What we don t know Will there be a regulation from the Government? How long will Big data change management. Will decisions be more and more based on data? Will Highest-paid person s opinion(hippo) no longer matter?

74 How Big Data will change your industry Industries that will be transformed Every industry that moves things. Every industry that sells to consumers. Every industry that employs machinery Every industry that sells/uses content. Every industry that provides services. Every industry that involves money

75 Future scenarios Lots of jobs (2 million) will be created to handle big data projects 4 million auxiliary jobs will also be created. Lots of data centers exist and more will come up.

76 Market Worldwide The current market in India and worldwide is enormous for big data. India: Current market: $1 billion. Huge opportunity worldwide as well.

77 Market Worldwide Wikibon: $5.1 billion in 2012 $53.4 billion in 2017 Wikibon is a community of practitioners and consultants on technology and business systems that uses open source sharing of free advisory knowledge.2007 headquartered in Marlborough, Massachusetts.

78 Topics to be covered Transformation due to big data Success factors Big data technology for managers What it takes to succeed with big data Learning from the experience of others What others are doing

79 The Challenge As a part of modeling and managing big data is focusing on the dimensions of scale availability and considering the challenges associated with this dimensions to pick the right tools.