A World of Data. Raghu Ramakrishnan. CTO for Data, Technical Fellow Microsoft

Size: px

Start display at page:

Download "A World of Data. Raghu Ramakrishnan. CTO for Data, Technical Fellow Microsoft"

Emil Harris
6 years ago
Views:

1 A World of Data Raghu Ramakrishnan CTO for Data, Technical Fellow Microsoft

minutes Dashboard (CORE) Provides real-time insights into performance by package, segment, and property Mix Management (Property) Ensures editorial voice is

3 Content Optimization Agrawal et al., CACM 56(6): (2013) Content Recommendation on Web Portals Key Features Package Ranker (CORE) Ranks packages by expected CTR based on data collected every 5 minutes Dashboard (CORE) Provides real-time insights into performance by package, segment, and property Mix Management (Property) Ensures editorial voice is maintained and user gets a variety of content Package rotation (Property) Tracks which stories a user has seen and rotates them after user has seen them for a certain period of time Key Performance Indicators Recommended links News Interests Top Searches Lifts in quantitative metrics Editorial Voice Preserved

4 Estimate P(response user, item, context) Statistical Methods for Recommender Systems, Agarwal and Chen, CUP, 2016

event streams) Online Learning Online regression models, time-series models Model the temporal dynamics Provide fast learning for per-item

5 Modeling Overview Offline Modeling Exploratory data analysis Regression, feature selection, collaborative filtering (factorization) Seed online models & explore/exploit methods at good initial points Reduce the set of candidate items Large amount of historical data (user event streams) Online Learning Online regression models, time-series models Model the temporal dynamics Provide fast learning for per-item models Near real-time user feedback Explore/Exploit Multi-armed bandits Find the best way of collecting realtime user feedback (for new items)

6 Store any data relations Data to Intelligent Action Do any analysis SQL queries Hive, At any speed Batch Hive At any scale elastic! Anywhere

Xbox Live Exchange STB Commerce Risk Messenger LCA Windows Phone Microsoft Stores Windows CRM/Dynamics SMSG STB Malware Protection Ads Live Office365

7 Xbox Live Exchange STB Commerce Risk Messenger LCA Windows Phone Microsoft Stores Windows CRM/Dynamics SMSG STB Malware Protection Ads Live Office365 Skype Bing Yammer # internal developers: 1000s # daily jobs: 100s of Ks daily I/O: >100 PBs data managed: EBs cluster sizes: 10s of Ks # machines: 100s of Ks

8 Azure Data Lake 1 st party = 3 rd party Hadoop and OSS

9 Our Axes of Innovation Data Operational Analytics SaaS Cloud IaaS PaaS In-Memory Data Lake Document Relational Intelligence (Algorithms, IoT )

10 Azure

11 34 Azure regions 2x as AWS

12 More certifications than any other cloud provider Industry leader for customer advocacy and privacy protection Unique data residency guarantees

13 Management Applications + App Frameworks Databases & Middleware Infrastructure Linux

14 Microsoft Contributions to OSS Apache YARN Amoeba Rayon Status: shipping in Apache Hadoop 2.6 Mercury and Yaq Status: Now in Apache Hadoop trunk! Federation Status: prototype and JIRA Framework-level Pooling Enable frameworks that want to take over resource allocation to support millisecondlevel response and adaptation times Status: spec

15 Hybrid Management + Security Log analytics Automation Backup DR and data protection Security

16 MQ leader quadrants Competitor 1 Competitor 2 Competitor 3

18 Cortana Intelligence Suite

19 Data and Analytics 3 Pillars Cortana Intelligence Suite Azure Big Data and Analytics Hadoop, Data Lake, Machine learning, PowerBI, Data Factory, Streaming, Perceptual Intelligence On-prem connectivity SQL 2016 Server Azure DB Azure DW SQL server R services On-prem and cloud (Windows, Linux) Microsoft R server Analytics Hadoop Teradata On-prem and cloud (Windows, Linux)

20 Forrester Wave Big Data Hadoop Cloud Solutions Q2 2016

23 Personalized Offers

24 DATA SOURCES INGEST PREPARE ANALYZE PUBLISH CONSUME In-Store Activity with Kinect sensors Diagnostic streaming Machine Learning Event hubs Stream Analytics Social Data Stream Analytics Cortana Azure SQL Data Warehouse Power BI On-Premise SQL Server (customer and product data) HDInsight HDInsight Machine Learning Azure DataLake Web/LOB Dashboards Web Logs, Omniture logs Data Factory: Move Data, Orchestrate, Schedule, and Monitor

25 Churn Prediction Tacoma Public Schools wanted to leverage data to predict student dropout risks to increase graduation rates

Personalized Healthcare Data from sensors and devices such as blood-pressure cuffs and activity trackers Cortana Analytics dashboard where registered nurses have a singular view of each customer s

26 Personalized Healthcare Data from sensors and devices such as blood-pressure cuffs and activity trackers Cortana Analytics dashboard where registered nurses have a singular view of each customer s personalized care plan Many of the solutions currently on the market give physicians access to raw data; that s not as useful as actionable intelligence to help them make a diagnosis. When you start looking at tools such as ImagineCare that have intelligence built in, I think that s a big deal for providers. DR. ETHAN BERKE Medical Director for Clinical Design and Innovation

To detect the small trending or level changes early in order to start timely investigations and actions to prevent

27 Service Analytics To replace manual threshold method to monitor the dynamics telemetry data to detect anomalies intelligently. To detect the small trending or level changes early in order to start timely investigations and actions to prevent potential incidents; To learn automatically from both historical and real-time data to scale the monitoring. SQL Azure uses Anomaly Detection models to track hundreds different service exceptions that won t be able to tracked just by setting thresholds.

28 Revenue Forecasting This helps us triangulate internal forecasts and gives us more confidence in the forward looking revenue ranges we provide to Wall Street. VANDANA GANGAWAR Senior Director, Microsoft Central Finance Planning and Operations

29 Azure Data Lake Store Fully managed cloud data store designed for analytics Supports HDFS compliant analytics applications and tools Petabyte files, unlimited account size High throughput for analytics performance Low latency ingestion with read as you write AAD-based authentication, access auditing File and folder-level ACLs, Encryption at rest

Pay PER QUERY & Scale PER QUERY - No need to create a cluster No limits to SCALE Azure Data Lake Analytics An elastic analytics service built on Apache YARN that processes all data, at any size

30 Pay PER QUERY & Scale PER QUERY - No need to create a cluster No limits to SCALE Azure Data Lake Analytics An elastic analytics service built on Apache YARN that processes all data, at any size Includes U-SQL, unifying the benefits of SQL with the expressive power of C# - In future: Hive, Spark Optimized to work with ADL STORE FEDERATED QUERY across Azure data sources ENTERPRISE GRADE role-based access control and auditing

32 Azure HDInsight Linux and Windows Managed, Monitored, Supported Cluster customization Install your favorite project Harness existing.net & Java skills to write customer extensions Supports broad ecosystem of ISVs (Hadoop and Traditional) Full Apache Hadoop Batch MapReduce, PIG, Hive, Spark Stream Processing and Analytics Storm, SparkStreaming Interactive SQL Hive (Tez), and SparkSQL Table Serving Hbase Machine Learning SparkML, Mahout

33 The Best of Hadoop Azure HDInsight Batch MapReduce, PIG, Hive, Spark Interactive SQL Hive (Tez), SparkSQL Stream Analytics Storm, SparkStreaming Machine Learning SparkML, Mahout Table Serving Hbase Exploratory Visualization Jupyter, Zeppelin Interactive SQL SQL DW Stream Analytics Azure Stream Analytics Machine Learning Azure ML Table Serving Azure SQL DB Exploratory Visualization Power BI

34 High-performance open source R plus: DeployR ConnectR ScaleR DistributedR DevelopR Enterprise Scale & Performance Scales from workstations to large clusters Scales to large data sizes Growing portfolio of Parallelized algorithms Secure, Scalable R Deployment/Operationalization Write Once Deploy Anywhere for multiple platforms IDE for data scientists and developers Enterprise Class Support

35 Hadoop EDW Hortonworks Cloudera MapR Teradata, SQL Server DevelopR ConnectR ScaleR DistributedR Workstations & Servers In the Cloud Linux Windows Azure VM Azure HDI, Spark Azure ML PowerBI Office 365 Code Portability Across Platforms

SQL Server 2016: Everything Built-In built-in built-in built-in built-in built-in 0-10 -20-30 -40 2010 2011 2012 2013 2014 2015 0 0 0 1 4 5 3 3 6 15 22 20 18 22 29 34 SQL Server SQL Server SQL Server

36 SQL Server 2016: Everything Built-In built-in built-in built-in built-in built-in SQL Server SQL Server SQL Server Microsoft Tableau Oracle $120 $ #1 #2 #3 Oracle is #4 $2,230 SQL Server Oracle MySQL2 SAP HANA TPC-H non-clustered 10TB Self-service BI per user at massive scale In-memory across all workloads Consistent experience from on-premises to cloud The above graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Microsoft. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose. National Institute of Standards and Technology Comprehensive Vulnerability Database update 5/4/2015 TPC-H non-clustered results as of 04/06/15, 5/04/15, 4/15/14 and 11/25/13, respectively.

37 In-Database Advanced Analytics No need to move the data Example Solutions Sales forecasting Warehouse efficiency Predictive maintenance Credit risk protection Extensibility R Integration R? New R scripts Microsoft Azure Marketplace Real-time operational NEW analytics without moving the data Analytic Library T-SQL interface Relational data R built-in to SQL Server Data Scientist Interact directly with data Data Developer/DBA Manage data and analytics together Open source R with inmemory & massive NEW scale multi-threading & massive parallel processing Mission critical OLTP End-to-end mobile BI Advanced Analytics

conversation Skype Consumer Groupme SMS (Twilio) Slack Language Understanding Image:

40 beginning with conversation, people & bots Conversation channels Active Directory Bot Securely access people in a company (+ files, topics, data) from anywhere, via conversation Skype Consumer Groupme SMS (Twilio) Slack Language Understanding Image: Face, Age, Gender, Emotion Academic Knowledge o365 apis Intelligence: Cognitive Services

Two offerings which interoperate really well

Microsoft Two offerings which interoperate really well On-premises Cortana Intelligence Suite SQL Server 2016 Cloud IAAS Enterprise PAAS Cloud Storage Service 9 SQL Server 2016: Everything built-in built-in