Big Data Introduction

Transcription

1 Big Data Introduction

2 Who we are Experts At Your Service Over 50 specialists in IT infrastructure Certified, experienced, passionate Based In Switzerland 100% self-financed Swiss company Over CHF8 mio. Turnover Leading In Infrastructure Services More than 150 customers in CH, D & F Over 50 SLAs dbi FlexService contracted Big Data Introduction Page 2

3 About me Mehdi Bada Consultant mehdi.bada[at]dbi-services.com Big Data Introduction Page 3

4 Agenda 1.Introduction 2.Big Data Analytics 3.Big Data standard architecture 4.Big Data lambda architecture 5.Big Data actors 6.Conclusion dbi services Page 4

5 Introduction 1 What s Big Data? Big Data ecosystem The Four V s of Big Data Big Data is Not Big Data Introduction Page 5

6 Introduction What s Big Data? Data Source - Open and private Data - IOT (Internet of things) - Social media - Customer behavior Data Visualization - Charts and Graphs - Javascript Libraries - Dashboards Big Data Storage and Processing - NoSQL databases - HDFS - Data Lake - Data Lab Analytics - Smart algorithms - Real time analysis - Batch processing - Lambda architecture Page 6

7 Introduction What s Big Data? No standard architecture available for Big Data use cases Choosing the right architecture depends on your needs The model has changed Old model Passive model Active model New model Page 7

8 Introduction Big Data ecosystem Page 8

9 Big Data Introduction The Four V s of Big Data Volume Velocity Variety Veracity Data at Scale Data in Motion Data in Many Forms Data in Doubt Page 9

10 Introduction Big Data is Not... Page 10

11 Big Data Analytics 1 Business Intelligence process Big Data process Big Data analytics challenges B.I & Big Data Big Data Introduction Page 11

12 Big Data Analytics Business Intelligence process Business requirements Old model Rigid model, Alteration, Long time Implementation Data source Data assessment Data cleansing Analyze Scope Collect Clean Visualization Page 12

13 Big Data Analytics Big Data process New model Fast and Flexible process Business requirements Data Data source source Data integration Data assessment Data exploration Data cleansing Business validation Analyze Process repetition Identification Semi/unstructured data Analyze, Patterns Implement solutions New sources, patterns Page 13

14 Big Data Analytics Big Data analytics challenges Distributed & multi server architecture Batch process Hardware improvement Diversified data Page 14

15 Big Data Analytics B.I & Big Data B.I and Big Data Data scientist and B.I analyst B.I Big Data B.I analyst Data scientist Data Source Altered RAW Goals KPIs, Reports Patterns, models Processing E.T.L Fast (In-memory) Actions Static Dynamic Analysis Static Dynamic Transform Slow, painful On-the-fly Visualization Dashboard Multi-support Analysis Past Predictive Page 15

16 Big Data standard architecture 1 Big Data process Summary Big Data Introduction Page 16

17 Big Data standard architecture Big Data process Data is generated more and more by many data sources devices and business) (human, Data Sources Page 17

18 Big Data standard architecture Big Data process Data ingestion, the best place to manage your cold / hot data Data Sources Data Ingestion Data at Rest Data Streaming Page 18

19 Big Data standard architecture Big Data process Data analysis gives you an enlightened view of your data Data Analysis Data Ingestion Data Lab Data Lake Page 19

20 Big Data standard architecture Big Data process Data processing is able to leverage data to drive business process Data Processing Data Analysis Batch Streaming/ Real-Time Query Engine Page 20

21 Big Data standard architecture Big Data process Data visualization allows you to have a customized view of your data Data Visualization Data Processing Marketing Supply Chain Sales Customers Page 21

22 Big Data standard architecture Summary Data Sources Data Ingestion Data Analysis Data Processing Data Visualization Data at Rest Data Lake Batch Reports Streaming data Data Lab Real-Time Analytics tools Predictive Page 22

23 Big Data standard architecture Summary Data Sources Data Ingestion Data Analysis Data Processing Data Visualization Page 23

24 Big Data lambda architecture 1 Lambda architecture Big Data Introduction Page 24

25 Big Data lambda architecture Lambda architecture A generic data processing architecture composed by 3 layers Speed Layer Streaming Processing Serving Layer New Data Streams Real- time Views Batch Views Query Batch Layer All Data Page 25

26 Big Data actors 1 Cloudera Microsoft Oracle Big Data Introduction Page 26

27 Big Data actors Cloudera Founded in 2008 Cloudera offers software for storage, access, manage, analysis, security and search CDH ecosystem: HDFS, MapReduce, Apache Spark, HBase, Kafka and Cloudera Search Cloud: AWS, Google Cloud, Microsoft Azure and Oracle Cloud Customers: Page 27

28 Big Data actors Microsoft Contribution to Hadoop: 10,000+ engineering hours Hadoop on Windows (Azure or On-premise) Collaboration between Microsoft and HortonWorks for Hadoop distribution Microsoft Analytics Platform System, a fully integrated system Spark for HDInsight provides connectors for BI tools such as Power BI and Tableau for data analytics Customers: Page 28

29 Big Data actors Oracle Oracle Big Data Appliance is the main component of the Oracle Big Data landscape Designed with Cloudera to deliver predictable Hadoop Oracle Big Data Discovery allows anyone to find, explore, transform and analyze Big Data sets Cloud: Oracle Big Data Cloud Machine, Oracle Big Data Cloud Services Compute Edition Customers: Wargaming.net, CaixaBank, Procter & Gamble, CERN Page 29

30 Conclusion 1 Big Data use cases Summary Big Data Introduction Page 30

31 Conclusion Big Data use cases Big Data is cool, but what else? Fraud detection (patterns recognition to detect anomalies) Trade risk (intraday analysis and historical analysis) Brand and feeling analysis (capture and processing direct feedback) Internet of Things (sensors, RFID chips ) Insurance (ability to assess the risk posed by a particular driver) Recruitment process (360 view of a candidate, ) Page 31

32 Conclusion Big Data use cases #RealMadrid Page 32

33 Conclusion Big Data use cases Created in ,800 processor cores 15 terabytes of RAM IBM supercomputer that combines artificial intelligence (AI) and sophisticated analytical software for optimal performance Hadoop ecosystem manages the task of preprocessing Watson s enormous information sources Page 33

34 Conclusion Summary Reduce Time to Market Many well known companies are major players Solutions are now mature and robust for production Data-driven enterprise can find new business opportunities Big Data can be used to develop the next generation of products and services Finding the right skills for Big Data projects Big Data infrastructure can be complex to implement Big Data reality is hidden by Marketing! Lack of resources to deploy, support Big Data infrastructure No operational standards available for Big Data technologies Page 34

35 Basel Delémont Zürich Any questions? Please do ask! Nyon We would love to boost your IT-Infrastructure How about you? Big Data Introduction Page 35