Do you remember the future? 2014 MapR Technologies 2

Transcription

1 2014 MapR Technologies 2014 MapR Technologies 1

2 Do you remember the future? 2014 MapR Technologies 2

3 2014 MapR Technologies 3

4 Some things turned out as expected 2014 MapR Technologies 4

5 Guys wearing Fedoras 2014 MapR Technologies 5

6 What about Big Data? 2014 MapR Technologies 6

7 Harvard University will have 200 x 10 6 volumes by 2040 Fremont Rider, MapR Technologies 7

8 To cope only short papers should be published. not more than 2500 characters counting space, punctuation marks, etc. Gray and Ruston in IEEE Transactions on Electronic Computers, MapR Technologies 8

9 Remember the guy in the Fedora? 2014 MapR Technologies 9

10 He s tweeting about this right now 2014 MapR Technologies 10

11 So for big data, What is the monorail? What is the Fedora? 2014 MapR Technologies 11

12 Data curation Rigid Schemas Engineered Structure 2014 MapR Technologies 12

13 Data curation Rigid Schemas Engineered Structure 2014 MapR Technologies 13

14 Data as-you-find-it Flexible schemas Late binding 2014 MapR Technologies 14

15 Data as-you-find-it Flexible schemas Late binding 2014 MapR Technologies 15

16 2014 MapR Technologies 16

17 2014 MapR Technologies 17

18 Fortune 100 Retailer 2014 MapR Technologies 18

19 Leading Cancer Research Center 2014 MapR Technologies 19

20 2014 MapR Technologies 20

21 Production Hadoop in Waste Management 2014 MapR Technologies 21

22 TREND Hadoop: The Disruptive Technology at the Core of Big Data JOB TRENDS FROM INDEED.COM Jan 06 Jan 07 Jan 08 Jan 09 Jan 10 Jan 11 Jan 12 Jan 13 Jan MapR Technologies 22

23 REALITY 2 Architecture Matters for Success FOUNDATION 2014 MapR Technologies 23

24 REALITY 2 Architecture Matters for Success NEW APPLICATIONS SLAs TRUSTED INFORMATION LOWER TCO Data protection & security Multi-tenancy Open standards for integration High performance Workload management FOUNDATION 2014 MapR Technologies 24

25 REALITY 3 Hadoop is Being Used to Drive Small, Rapid Decisions High Arrival Rate Data Clickstream Social media Sensor data, Business Impact Revenue optimization Risk mitigation Operational efficiency 2014 MapR Technologies 25

26 Sellers Cloud Advertising Automation Cloud Buyers Cloud 100B AD AUCTIONS per day 2014 MapR Technologies 26

27 Largest Biometric Database in the World 1.2B PEOPLE PEOPLE 2014 MapR Technologies 27

28 50M SET-TOP BOXES 2014 MapR Technologies 28

29 Fortune 100 Financial Services Company 104M CARD MEMBERS 2014 MapR Technologies 29

30 REAL So what is the future of Big Data 2014 MapR Technologies 30

31 Old future == map-reduce New future == no SQL 2014 MapR Technologies 31

32 Old future == map-reduce New future == no SQL (revised) 2014 MapR Technologies 32

33 ... revised!? What else is needed? 2014 MapR Technologies 33

34 SQL (yes, really) Limited transaction support Higher scan performance Integrated with file storage Snapshots/mirrors Incremental aggregates 2014 MapR Technologies 34

35 Why does nosql require SQL? Most queries are written by a combination of SQL speaking machines (Tableau and such) SQL speaking analysts Kinda, sorta SQL doesn t work for each audience Making either speak non-sql languages is boiling the ocean 2014 MapR Technologies 35

36 Transactions? Don t transactions kill performance and scaling? Yes General transactions can be very difficult to support at speed and scale But, no Coordinated read/modify/write of a handful of records at once Atomic incorporation of bulk inserts, upserts, updates 2014 MapR Technologies 36

37 Higher scan performance? Record oriented scan is hopelessly slow Column oriented storage, late materialization option required Raw scan rates can exceed 1GB/s/node for disk Specialist compression can allow effective processing at 20-50GB/s Raw memory rates can exceed 50GB/s/node for main memory Specialist compression can allow 20x higher processing Overall speedups available nearly 1000x 2014 MapR Technologies 37

38 Integrated with file storage? Not everything is a table Files make very good message busses Real integration requires valid references References imply limited joint linearizability Snapshots and mirrors expose update failures 2014 MapR Technologies 38

39 Snapshots and Mirrors Real file systems and real databases have done this for years Now considered table stakes Like version control for data 2014 MapR Technologies 39

40 Incremental Aggregates See Druid Can allow incremental performance for massive computations Real-time systems have the advantage for aggregates Can take 3 years to process 3 years of data 2014 MapR Technologies 40

41 But who? You! 2014 MapR Technologies 41

42 But who? Me! 2014 MapR Technologies 42

43 But who? Us! 2014 MapR Technologies 43