Jason Virtue Business Intelligence Technical Professional

Size: px

Start display at page:

Download "Jason Virtue Business Intelligence Technical Professional"

Donna Hodges
6 years ago
Views:

1 Jason Virtue Business Intelligence Technical Professional

2 Agenda Microsoft Azure Data Services Azure Cloud Services Azure Machine Learning Azure Service Bus Azure Stream Analytics Azure HDInsight Azure Websites

3 Cloud Data Sources

4 Expensive Siloed data Disconnected tools Deployment complexity Huge set-up costs of tools, expertise, and compute/storage capacity create unnecessary barriers to entry Siloed and cumbersome data management restricts access to data Complex and fragmented tools limit participation in exploring data and building models Many models never achieve business value due to difficulties with deploying to production The Cloud Changes the Landscape

5 Microsoft Azure Data Services Producers Data Transport Storage Analytics Presentation & action Event Hubs (Service Bus) SQL Database Machine Learning Azure Websites Heterogeneous client agents Table/Blob Storage HD Insight Mobile Services External Data Sources { } DocumentDB Stream Analytics Notification Hubs External Data Sources Cloud Services Power BI External Services

6 DATA SCIENCE is the new BUSINESS INTELLIGENCE Business Intelligence Data Science

Twitter Demo flow and Architecture Twitter Sentiment140

2 Filtered tweets Event Hub Inbound Tweets Stream

Aggregates Worker Role Tweet Consumer Data pushed to

Analytics Aggregate Tweets Event Hub Tweet Aggregates

7 Twitter Demo flow and Architecture Twitter Sentiment140 Worker Role Tweet Publisher 1 Worker Role Tweet Publisher 2 Filtered tweets Event Hub Inbound Tweets Stream Analytics Aggregate Tweets by Topics Event Hub Topic Aggregates Worker Role Tweet Consumer Data pushed to clients using SignalR Website Real-Time Dashboard Stream Analytics Aggregate Tweets Event Hub Tweet Aggregates Stream Analytics Archive Tweets Machine Learning Hive Blob Storage Map/Reduce HDFS Power BI

9 What is in a Cloud Service?

10 What Can It Run?

Microsoft & Machine Learning Answering questions with experience 1991 1997 2008 2009 2010 2014 2014 Microsoft Research formed Hotmail

email is junk? What s the best way home? Which searches are most relevant? What does that motion mean? What is that person saying?

12 Microsoft & Machine Learning Answering questions with experience Microsoft Research formed Hotmail launches Bing maps launches Bing search launches Kinect launches Skype Translator launches Azure Machine Learning launches Which is junk? What s the best way home? Which searches are most relevant? What does that motion mean? What is that person saying? What will happen next? John Platt, Distinguished scientist at Microsoft Research Machine learning is pervasive throughout Microsoft products.

PCs/ Laptops POS Terminals Self Checkout Stations Smart Phones Kiosks Slates/ Tablets Point of Service Devices Automation Devices Servers Digital

17 PCs/ Laptops POS Terminals Self Checkout Stations Smart Phones Kiosks Slates/ Tablets Point of Service Devices Automation Devices Servers Digital Signs Logic Controllers ATM Security Thin Clients Remote Medical Monitors Vending Machines Handhelds Kinect Specialized Devices Diagnostic Equipment

20 MapReduce Hive Pig C# Stored Procedures

21 Original Data Transformed Data Layer-Cake Approach Evolving Approaches to Analytics Traditional EDW Big Data Processing - Ingest first, refinement OnDemand BI Tools Data Marts BI Tools Data Marts Apps Dashboards EDW (SQL Svr, Teradata, etc) Ingest (EL) Original Data Scale-out Storage & Compute (HDFS, Blob Storage, etc) Load Streaming data ETL Tool (SSIS, etc) Transform Transform & Load Extract Dashboards Data Lake(s) Apps

23 Microsoft HDInsight Hadoop as a Service HBase as a columnar NoSQL transactional database running on Azure Blobs Storm as a streaming service for near real time processing Hadoop 2.4 support for 100x query gains on Hive queries Mahout support for machine learning + Hadoop Graphical User Interface for HIVE queries Coordination Support HBase as NoSQL columnar database on Azure Blobs Support Storm as stream processing HMaster Name Node Region Server Region Server Region Server Region Server Job Tracker Data Node Data Node Data Node Data Node Task Tracker Task Tracker Task Tracker Task Tracker Microsoft Confidential Under Strict NDA

24 Hadoop is a platform with portfolio of projects

25 A Hadoop distribution is a package of projects

26 Programmableweb.com Directory

2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

27 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

AZURE HDINSIGHT. Azure Machine Learning Track Marek Chmel

AZURE HDINSIGHT. Azure Machine Learning Track Marek Chmel AZURE HDINSIGHT Azure Machine Learning Track Marek Chmel SESSION AGENDA Understanding different scenarios of Hadoop Building an end to end pipeline using HDInsight Using in-memory techniques to analyze