HDInsight - Hadoop for the Commoner Matt Stenzel Data Platform Technical Specialist

Transcription

1 HDInsight - Hadoop for the Commoner Matt Stenzel Data Platform Technical Specialist

2 SQL Saturday #557 Thank you Sponsors! Please visit the sponsors and enter their end-of-day raffles. Event After Party Sky Deck Sports Grille and Lanes at the Mall of America at 7 PM. Want More Free Training? PASSMN meets the 3rd Tuesday of every month Matt Stenzel HDInsight Hadoop for the Commoner

3 Lunch Sponsor - Dell EMC For those who paid for lunch already, we will refund you via PayPal. If you wish to donate to Rebecca CoderDojo, please drop your ticket in the bucket at registration Matt Stenzel HDInsight Hadoop for the Commoner

4 You Rock Sponsor - Pyramid Analytics Gold Sponsors IDERA Pragmatic Works VMWare GNet Tail Wind Microsoft Dell Software Matt Stenzel HDInsight Hadoop for the Commoner

5 Other Sponsors Silver Sponsors Improving Experts Exchange Pure Storage Bronze Sponsors SQL Sentry COZYROC PASS Blog Sponsors SQLVariant Matt Stenzel HDInsight Hadoop for the Commoner

6 Who ARE you? Matt Stenzel Background Developer DBA/Hadoop Admin Architect Data Platform Technical Specialist Microsoft SQL Data Warehousing Big Data Analytics Contact Matt Stenzel HDInsight Hadoop for the Commoner

7 WHY ARE YOU HERE? Matt Stenzel HDInsight Hadoop for the Commoner

8 Session Objectives and Takeaways Why Hadoop? What is Hadoop? Why HDInsight? What is HDInsight? Provision HDInsight Matt Stenzel HDInsight Hadoop for the Commoner

9 SPEAKER CHALLENGE PROVISION AND USE HADOOP CLUSTER IN AN HOUR DEMO Matt Stenzel HDInsight Hadoop for the Commoner

10 SPEAKER CHALLENGE ACCEPTED Matt Stenzel HDInsight Hadoop for the Commoner

11 DEMO! - PROVISION HDINSIGHT Matt Stenzel HDInsight Hadoop for the Commoner

12 WHY HADOOP? Matt Stenzel HDInsight Hadoop for the Commoner

13 Breaking points of traditional approach

14 Hadoop Scenario 1: pre-process ETL

15 Hadoop Scenario 2: hot and cold storage Hot data in DW Cold data in Hadoop

16 Hadoop Scenario 3: true data discovery

17 WHAT IS HADOOP? Matt Stenzel HDInsight Hadoop for the Commoner

18 Apache Hadoop

19 Hadoop is a platform with portfolio of projects

20 Data volume

21 Data variety

22 Data velocity

23 Business applications of Hadoop

24 HADOOP CHALLENGES Matt Stenzel HDInsight Hadoop for the Commoner

25 Challenges with implementing Hadoop

26 Why Hadoop in the Cloud?

27 WHAT IS HDINSIGHT? Matt Stenzel HDInsight Hadoop for the Commoner

28 Cortana Intelligence Suite Data Sources Information Management Big Data Stores Machine Learning and Analytics Intelligence People Data Factory Data Lake Store Machine Learning Cognitive Services Data Catalog SQL Data Warehouse Data Lake Analytics Bot Framework Web Apps Event Hubs HDInsight (Hadoop and Spark) Cortana Apps Mobile Stream Analytics Bots Dashboards & Visualizations Sensors and devices Power BI Automated Systems Data Intelligence Action

29 Azure HDInsight

30 Microsoft contributions to Hadoop

31 HDInsight Built for Windows or Linux

32 HDInsight Supports Hive

33 HDInsight Supports HBase Coordination HMaster Name Node Region Server Region Server Region Server Region Server Job Tracker Data Node Data Node Data Node Data Node Task Tracker Task Tracker Task Tracker Task Tracker

34 HDInsight Supports Mahout

35 HDInsight Supports Storm RabbitMQ / ActiveMQ Web/thick client dashboards Stream processin g Search and query Data analytics (Excel) Devices to take action

36 Spark for Azure HDInsight In Memory Processing on Multiple Workloads Single execution model for multiple tasks Processing up to 100x faster performance Developer friendly (Java, Python, Scala) Azure HDInsight In Memory Spark BI tool of choice (Power BI, Tabelau, Qlik, SAP) Notebook experience (Jupyter/iPython, Zeppelin)

37 R Server for HDInsight Familiarity of R (most popular language for data scientists) Scalability of Hadoop and Spark Up to 7x faster using Spark engine Train and run ML models on datasets of any size Cloud managed solution (easy setup, elastic, SLA)

38 HDInsight Allows You To Add Hadoop Projects

39 Making advanced analytics accessible to Hadoop Cloud

40 DEMO USING HDINSIGHT Matt Stenzel HDInsight Hadoop for the Commoner

41 Recap Provision and Scale HDInsight Why Hadoop? What is Hadoop? Why HDInsight? What is HDInsight? Matt Stenzel HDInsight Hadoop for the Commoner

42 Get Started Read documentation Learning Map Microsoft Virtual Academy Channel 9 Data Exposed Show Try 30 day trial Matt Stenzel HDInsight Hadoop for the Commoner

43 Get Started Edx Courses Matt Stenzel HDInsight Hadoop for the Commoner

44 Contact Info Matt Matt Stenzel HDInsight Hadoop for the Commoner

45 PolyBase for SQL Server 2016

46 PolyBase View in SQL Server 2016 PolyBase View Execute T-SQL queries against relational data in SQL Server and semi-structured data in Hadoop or Azure Blob Storage Leverage existing T-SQL skills and BI tools to gain insights from different data stores Access any data

47 PolyBase use cases Access any data

48 PolyBase Demo