Stream Analytics for a SQL Developer. Slava Trofimov SQL Saturday - Cincinnati, OH March 17, 2018

Size: px

Start display at page:

Download "Stream Analytics for a SQL Developer. Slava Trofimov SQL Saturday - Cincinnati, OH March 17, 2018"

Kristian Reynard O’Brien’
5 years ago
Views:

1 Stream Analytics for a SQL Developer Slava Trofimov SQL Saturday - Cincinnati, OH March 17, 2018

2 About Me Practice Lead, Data Platform & Analytics, KiZAN Technologies 15 years of experience in data management, BI and analytics 2

3 What is Stream Analytics? Continuous, real-time analysis of data in motion. Sensor data, financial transactions, clickstream data, web searches, network traffic, IoT data, etc. 3

4 Why Real-Time Analytics? Data is perishable: the value of certain data diminishes rapidly over time Data is more actionable when it is fresh Pre-processing of data in real time may facilitate subsequent historical analysis 4

5 Real-Time Analytics Scenarios Customer & KiZAN confidentl Source: Microsoft 5

6 Why Stream Analytics in the Cloud? Event data is geographically distributed Event data may already be in the cloud Infrastructure required to build scalable, resilient and feature-rich real time analytics solutions is complex and expensive Focus on solutions, not infrastructure 8

7 Cloud Overview: IaaS, PaaS, SaaS Source: Nick Landry, Microsoft 9

8 What is Azure Stream Analytics? Managed Service Real-time stream processing in the cloud Scale dynamically: up to millions of events per second Mission-critical reliability Rapid development and deployment Integrated with Azure data platform, Azure Machine Learning and big data analytics services Source: Microsoft 10

9 Cortana Intelligence Suite A fully managed big data and advanced analytics suite to transform your data into intelligent action. Source: Microsoft 11

Architecture Overview Azure Stream Analytics Event Inputs - Event Hub - IoT Hub - Azure Blob Reference Data - Azure Blob Transform - Temporal joins - Filter - Aggregates - Projections - Windows -

10 Architecture Overview Azure Stream Analytics Event Inputs - Event Hub - IoT Hub - Azure Blob Reference Data - Azure Blob Transform - Temporal joins - Filter - Aggregates - Projections - Windows - Machine Learning Functions - Etc. Enrich Correlate Outputs - SQL Database - SQL Data Warehouse - Blob Storage - Table Storage - Service Bus - CosmosDB - Event Hub - IoT Hub - Data Lake Store - Power BI Azure Storage Source: Microsoft Data Source Collect Process Deliver Consume

11 Streaming Data Analysis Concepts Data is in motion Near real time Data arrives temporally as a stream of events The stream is potentially infinite You can t aggregate data all at once your analysis never ends! Data is grouped into time windows 13

12 Temporal Windowing Tumbling Window Hopping Window Sliding Window Source: MSDN 14

13 Tumbling Window Fixed-length time windows Non-overlapping and contiguous Event can belong to only one tumbling window TUMBLINGWINDOW(timeunit, windowsize) Source: MSDN 15

14 Hopping Window Fixed-length time windows Window size may be different from hop size Windows may overlap Event can belong to multiple hopping windows HOPPINGWINDOW ( timeunit, windowsize, hopsize ) Source: MSDN 16

15 Sliding Window Fixed-length time windows Considers all possible windows of a given length Outputs an event when content of window changes (e.g. event entered or exited the window) No output when the window is empty Event can belong to multiple sliding windows SLIDINGWINDOW ( timeunit, windowsize ) Source: MSDN 17

16 Topology of a Stream Analytics Job Mandatory Components One or more inputs One or more outputs A query to transform data from Inputs to Outputs Optional Components Machine Learning Functions JavaScript User Defined Functions JavaScript User Defined Aggregates 18

Temporal semantics All operators respect, and some use, the temporal properties of events Built-in operators and functions

17 Declarative SQL-like queries No code compilation, easy to author and deploy Unified programming model Brings together event streams, reference data and machine learning extensions, and user defined functions. Temporal semantics All operators respect, and some use, the temporal properties of events Built-in operators and functions Filters, projections, joins, windowed (temporal) aggregates, text and date manipulation Support for complex data types Ability to process and manipulate records and arrays.

18 Stream Analytics Query Language Data Manipulation Language SELECT, FROM, WHERE INTO GROUP BY HAVING, WITH, CASE, INNER/LEFT JOIN, UNION APPLY, ORDER BY Scaling Extensions PARTITION BY, WITH Time Management TIMESTAMP BY TUMBLINGWINDOW HOPPINGWINDOW SLIDINGWINDOW Analytic Functions ISFIRST LAG, LAST Aggregate Functions AVG, SUM, COUNT, MAX, MIN, STDEV, STDEVP, VAR, VARP TopOne, Collect, CollectTOP Conversion Functions CAST, TRY_CAST GetType Date and Time Functions DATENAME, DATEPART, DAY, MONTH, YEAR, DATETIMEFROMPARTS, DATEDIFF, DATEADD Mathematical Functions ABS, CEILING, EXP, FLOOR, POWER, SIGN, SQUARE, SQR Record Functions GetRecordProperties, GetRecordPropertyValue Array Functions GetArrayElement, GetArrayElements, GetArrayLength String Functions LEN, CONCATE, CHARINDEX, LOWER, SUBSTRING, PATINDEX, UPPER GeoSpatial Functions CreatePoint, CreateLineString, CreatePolygon, ST_DISTANCE, ST_OVERLAPS, ST_INTERSECTS, ST_WITHIN 20

19 Sample Streaming Record { } "eventenqueuedutctime": " T22:00: Z", "deviceid": "kizan-device-10", "tempc": 20.39, "tempf": 68.7, "runtime": 14662, "EventProcessedUtcTime": " T22:00: Z", "PartitionId": 1, "IoTHub": { "MessageId": null, "CorrelationId": null, "ConnectionDeviceId": "kizan-device-10", "ConnectionDeviceGenerationId": " ", "EnqueuedTime": " T21:59: Z", "StreamId": null } 21

20 Demo 22

21 Scenario for Today s End-to-End Demo Real-time predictive maintenance solution 1. Collect temperature data from electrical motors 2. Send telemetry data to the Azure cloud 3. Process telemetry data in real time and predict potential failures 4. Visualize telemetry data in real time 5. Store telemetry data for future analysis 6. Perform advanced analysis and visualization of historical data 26

22 Solution Architecture

23 Thoughts? Questions? Ideas? 30

24 Resources Azure Stream Analytics Introduction: Guided learning path: Power BI Power BI Guided Learning: Self-paced edx course: 31

Boston Azure Cloud User Group. a journey of a thousand miles begins with a single step

Boston Azure Cloud User Group. a journey of a thousand miles begins with a single step Boston Azure Cloud User Group a journey of a thousand miles begins with a single step 3 Solution Architect at Slalom Boston Business Intelligence User Group Leader I am a bit shy but passionate. BI Architect