Ray M Sugiarto MAPR Champion Indonesia

Transcription

1 Ray M Sugiarto MAPR Champion Indonesia MapR Technologies 2015 MapR Technologies 1

2 Why Big Data? University of Texas: The median Fortune 1000 company could increase its revenue by more than $2 billion a year if it increases data usability by 10 percent. Forbes, 12/4/ MapR Technologies 2

3 Market Outlook Industrial Internet Insights for 2015, from GE (NYSE: GE) and Accenture (NYSE:ACN) 2015 MapR Technologies 3

4 Fastest Adoption of New Enterprise Technology Large mission-critical, operational deployments Hadoop trials, science projects in a corner 2015 MapR Technologies 4

5 Leading Data Driven Customers Empowering the As-it-happens business by speeding up the data-to-action cycle 2015 MapR Technologies 5

6 Customer Case Studies MapR MapR Technologies 6

7 Largest Biometric Database in the World 1.2B PEOPLE PEOPLE MapR Technologies 7

8 Leading the Automation of Advertising 330 MapR Hadoop nodes 300 DECISIONS per transaction 100B+ BID REQUESTS per day < 80ms per transaction Millions QUERIES per second 20K MPBS of data 3M MEDIA FILES SCANNED 2.5M PEAK IMPRESSIONS per second 2015 MapR Technologies 8

9 Analytics for a Digital World 400 MapR Hadoop nodes 20TB NEW DATA INGESTED per day < 720,000 DIGITAL INTERACTIONS COLLECTED per second 10X COMPUTATIONAL SPEED 1.7T RECORDS PROCESSED per month MapR Technologies 9

10 FIN SERVICES Offer Serving, Credit Risk & Fraud 45s TERASORT 1.6TB MINUTESORT MapR Hadoop nodes Largest deployment in financial services 100M+ CARDS GOAL: 10Years DATA STORED < $ 900B WORLDWIDE BILLS $100M SAVED FOR CARDHOLDERS 2015 MapR Technologies 10

11 TOP 5 RETAILER W O R L D W I D E Targeted Marketing: In-store Geo-located Offers MapR Hadoop nodes Largest deployment in retail 200 DATA SCIENTISTS 245M CUSTOMERS per week +2% CONVERSION RATE IMPROVEMENT 40TB per NODE 7PB per CLUSTER +50 PRODUCTION APPLICATIONS MapR Technologies 11

12 Manage and Adapt to Climate Change 85% OF FARMER RISK IS WEATHER RELATED 10T DATA POINTS < 100TB DATA Corn, weat growing from 10K 2.5M OUTCOMES per location SENSORS 60Yrs 2M CROP-YIELD statistics LOCATIONS Natl. Weather Service Doppler Scans 2015 MapR Technologies 12

13 Entertaining Millions 2015 MapR Technologies 13

14 Machine Zone Topology Gamers Gaming servers + MySQL data copying Analytics with Hadoop Before Remote data center Local data center After Gamers Gaming servers + MapR-DB table mirroring Analytics with Enterprise Database Edition Remote data center Local data center 2015 MapR Technologies 14

15 Lessons to Apply 2015 MapR Technologies 15

16 Apps dictated the data format Data freely supports varied compute engines Data Silos Can t have silos ETL Ingest & go 2015 MapR Technologies 16

17 Lesson #1 Real-Time Requires Big AND Fast (in one cluster) 2015 MapR Technologies 17

18 Jul 2009 Agust 2009 Sep 2009 Okt 2009 Nop 2009 Des 2009 Jan 2010 Feb 2010 Mar 2010 Apr 2010 Mei 2010 Jun 2010 Jul 2010 Agust 2010 Sep 2010 Okt 2010 Nop 2010 Des 2010 Jan 2011 Feb 2011 Mar 2011 Apr 2011 Mei 2011 Jun 2011 Jul 2011 Agust 2011 Sep 2011 Okt 2011 Nop 2011 Des 2011 Jan 2012 Feb 2012 Mar 2012 Apr 2012 Mei 2012 Jun 2012 Jul 2012 Agust 2012 Sep 2012 Okt 2012 Nop 2012 Des 2012 Jan 2013 Feb 2013 Mar 2013 Apr 2013 Mei 2013 Jun 2013 Jul 2013 Agust 2013 Sep 2013 Okt 2013 Nop 2013 Des 2013 Jan 2014 Feb 2014 Mar 2014 Apr 2014 Mei 2014 Jun 2014 Jul 2014 Agust 2014 Sep 2014 # of records comscore Big Data Expansion Total records collected in September 2014 = 1,758,229,317,769 Total records collected YTD 2014 = 15,423,470,100, Billion Beacon Records Panel Records Billion Billion Billion Billion Billion 800 Billion 600 Billion 400 Billion 200 Billion Billion 2015 MapR Technologies 18

19 Big Data Drivers: Cost Efficiencies IT Budget vs Data Growth Storage Cost terabyte $40,000 $9,000 <$1, ENTERPRISE STORAGE DATABASE WAREHOUSE Gartner, "Forecast Analysis: Enterprise IT Spending by Vertical Industry Market, Worldwide, , 3Q12 Update. Wall Street Journal, Financial Services Companies Firms See Results from Big Data Push, Jan. 27, MapR Technologies 19

20 More DATA beats complex algorithms The Unreasonable Effectiveness of Data, published by Google 2015 MapR Technologies 20

21 More Data Allows You to Spot Infrequent Behaviors f With recent data you have limited historical data accumulated t t+1 t+2 Time (t=years) t+n MapR Technologies 21

22 More Data Allows You to Spot Infrequent Behaviors f With big data, you can trace infrequent patterns through time that call out anomalies. t t+1 t+2 t+n Time (t=years) MapR Technologies 22

23 Fast Data: Performance with Low Latency 2015 MapR Technologies 23

24 Typical Hadoop/DBMS Integration Churn analysis (predictive analytics) Customer 360 dashboard Data exploration (SQL) Mobile application server Web application server Analytics Operational Hadoop Batch import/export Operational DBMS (NoSQL or RDBMS) 2015 MapR Technologies 24

25 A Better Hadoop/DBMS Integration MapR-DB Real-time ad targeting Product/service optimization and personalization Churn analysis (predictive analytics) Customer 360 dashboard Data exploration (SQL) Mobile application server Web application server Real-Time and Actionable Analytics User profiles and state User interactions Real-time location data Web and mobile session state Comments/rankings Operational MapR Distribution with MapR-DB 2015 MapR Technologies 25

26 Alternatives Do Big OR Fast, But Not Both Limited reliability Limited functionality Batch (mostly) Rewrite existing apps Big OR Fast Expensive Special purpose Coding required for speed Limited scale Limited functionality 2015 MapR Technologies 26

27 For Real-time It Has To Be Both Big And Fast Unlimited Scale Data diversity Evolving schemas Affordable Secure Reliable Manageable Big AND Fast Fast ingestion Batch & streaming Polyglot RDBMS & NoSQL General purpose Affordable 2015 MapR Technologies 27

28 Lesson #2 Data Agility is a Must Managing it Processing it Exploring it Using it 2015 MapR Technologies 28

29 Data Agility Infrastructure Issues Traditional Approaches Business (analysts, developers) Plumbing development and data structuring Data Data Agility Business (analysts, developers) Data 2015 MapR Technologies 29

30 Self-Service Data Exploration Single SQL Interface for Structured and Semi-Structured Data Data Agility with Less IT Required 2015 MapR Technologies 30

31 Apache Drill Brings Flexibility & Performance Access to any data type, any data source Relational Nested data Schema-less Rapid time to insights Query data in-situ No Schemas required Easy to get started Integration with existing tools ANSI SQL BI tool integration Scale in all dimensions TB-PB of scale 1000 s of users 1000 s of nodes Granular security Authentication Row/column level controls De-centralized 2015 MapR Technologies 31

32 Drill s Role in the Enterprise Data Architecture Raw data Optimized data Centrally-structured data Relational data JSON, CSV,... Parquet, Schemas in Hive Metastore Highly-structured data Exploration (known and unknown questions) Oracle, Teradata Hive, Impala, Spark SQL 2015 MapR Technologies 32

33 Advice for your Journey No matter what use case you start with you will require real-time & enterprise-grade As it happens is just as much about business process reinvention as technology 2015 MapR Technologies 33

34 2015 MapR Technologies 2015 MapR Technologies 34

35 Enterprise Data Optimization MapR MapR Technologies 35

36 Data Warehouse Optimization Offloading cold data to Hadoop Restores Storage capacity One-time offload capitalizes on historic underused data Minimal impact to existing data pipelines Data Transformation/ETL on Hadoop Present new data for exploration ETL work includes incremental updates Restores CPU capacity and storage to DW 2015 MapR Technologies 36

37 Offload Cold Data to Hadoop Log Archive Incoming Data Data Warehouse ETL Structured Data Data Query-able Inexpensive Bulk Restores DW Process: One-time Migration Standard Apache Tools MapR Data Platform Cold Data Offload Data Access: ODBC Thrift, REST Standard Connectors 2015 MapR Technologies 37

38 ETL in Hadoop Data Warehouse Restored CPU and Disk Low Latency Data I want to get all this off so I can use it again Restores even more CPU and Disk Incoming Data ETL MapR Bulk Data Improves old DW Response and Speed Most Cost Efficient CPU and Storage 2015 MapR Technologies 38

39 Recommendation Engine MapR MapR Technologies 39

40 The Scenario Users constantly interact with items LOTS of users; LOTS of reusable items Relationships among items persist in the short term, while user behavior can change instantly The more types of interactions available, the better Interactions / preferences need not be explicit ratings Recommendations are needed in real-time your products here 2015 MapR Technologies 40

41 Building the Use Case purchases, views, etc. scrolls, clicks, etc. family history, diagnosis customers purchases products visitors patients U S E R S preferences I T E M S articles / content diseases cardholders anything sell more products V A L U E increase subscriptions, ads, etc. predict future illness detect anomalous purchases 2015 MapR Technologies 41

42 Recommendation Engine Workflow New User History (past affiliations) Recommendations (based on the indicators of affiliated items) Web Tier O N L I N E Item Meta-Data Data Ingest SQL Index User Histories Ingest Pre- Process Machine Learning Update Documents Post- Process O F F L I N E MapR Cluster 2015 MapR Technologies 42

43 Fraud & Anomaly Detection MapR MapR Technologies 43

44 Fraud & Anomaly Detection Real-time Operational Applications Online transactions Fraud detection Fraud model Personalized offers Recommendations MapR Hadoop Distribution Analytics Fraud investigator Interactive marketer Fraud investigation tool Clickstream analysis 2015 MapR Technologies 44

45 Fraud & Security Analytics Architecture Data Sources Firewall/Proxy Logs IDS/IPS Logs Server Logs Application Logs Identity access Logs Resource Access Logs Privileged user activity logs Database logs Miscellaneous Logs Security Feeds Sqoop, NFS Ingest, ETL, Batch Processing Hive, Pig Interactive SQL Drill MAPR DISTRIBUTION FOR MapR Data Platform HADOOP Realtime Processing Spark Streaming, Mahout, MLLib, SparkSQL Operational NFS MapR-FS MapR-DB MapR-FS HBase API SIEM/Splunk Analytics Search Schema-less data exploration Compliance reporting Ad-hoc integrated analytics Operational Apps Anomaly/Threat Detection Fraud Detection 2015 MapR Technologies 45

46 Techniques Anti-Money Laundering System Consumer Transactions Data Lake (Hadoop) Suspicious Events Analyst Latent Dirichlet Allocation Trained Classifier T-digest Percentiles Clustering / Peer Group Analysis Markov Models 2015 MapR Technologies 46

47 Visualization of Techniques alert Clustering Find closest cluster accumulate By Host threshold New point Reservoir Find closest sample alert Disappearing periodic event due potentially to malware 2015 MapR Technologies 47

48 What Does Big Data Mean to You? Customer stats on how MapR customers are using it 63% 40% = 21% Cost Reduction Revenue Risk Reduction 2015 MapR Technologies 48

49 What Does Big Data Mean to You? Customer stats on business impact (ROI) of MapR 65% 40% 10% 5% More than 5x More than 10x More than 20x More than 25x 2015 MapR Technologies 49

50 Top-Ranked Hadoop Distribution Top-Ranked NoSQL Top-Ranked SQL-on-Hadoop Solution 2015 MapR Technologies 50

51 Free Hadoop On-Demand Training $50M In-Kind Contribution to the Hadoop Community MapR Technologies 51

52 Q & A Ray Sugiarto MAPR Champion Indonesia maprtech mapr-technologies MapR bsadeghi@mapr.com maprtech 2015 MapR Technologies 52