Boston Azure Cloud User Group. a journey of a thousand miles begins with a single step

Size: px

Start display at page:

Download "Boston Azure Cloud User Group. a journey of a thousand miles begins with a single step"

Gladys Fitzgerald
6 years ago
Views:

3 Boston Azure Cloud User Group a journey of a thousand miles begins with a single step 3

4 Solution Architect at Slalom Boston Business Intelligence User Group Leader

Enterprise Business Intelligence Systems,

5 I am a bit shy but passionate. BI Architect Speaker Mentor Business Intelligence Architect, Mentor and Speaker. I am specializing in Design and Development of the Enterprise Business Intelligence Systems, Intelligent Applications, Real-Time and Big Data Analytic solutions. For the past 8 years as Boston Business Intelligence User Group Leader vkokaev@gmail.com

6 What is Stream Analytics and why to use it Real-time processing applications Usage, Scalability, Reliability and Cost Brief introduction to event hubs Configure, Develop and Deploy Streaming Solution Performing transformations and computations over streams of events 7

7 From todays presentation 8

8 1

9 Microsoft Azure Stream Analytics makes it easy to set up real-time analytic computations on data streaming from devices, sensors, web sites, social media, applications, infrastructure systems, and more 10

10 11

12 Businesses that can make better, faster decisions in response to their customers or operations stand to gain market share by delivering higher levels of customer satisfaction, higher quality, and ultimately obtaining larger wallet share. The ability to exercise rapid response rates to business events in seconds, rather than minutes or hours can yield significant revenue upside to company operations. 13

13 Vast amounts of data are flowing at high velocity over the wire today. Organizations that can process and act on this streaming data in real time can dramatically improve efficiencies and differentiate themselves in the market. Scenarios of real-time streaming analytics can be found across all industries: 14

14 6 points, icons and descriptions Stream Analytics connects directly to Azure Event Hubs for stream ingestion, and the Azure Blob service to ingest historical data. Stream Analytics supports a simple, declarative query model for describing transformations. ASA provides users the ability to specify reference data and join it with event streams ingested in real time to perform transformations. Stream Analytics is optimized to provide users a very low cost to get going and maintain real-time analytics solutions. Stream Analytics is capable of handling high event throughput of up to 1GB/second. ASA prevents data loss and provides business continuity in the presence of failures through built-in recovery capabilities. 15

15 2

16 An airplane data Sensors monitors the health of critical devices Airplane have thousands of parts that have to be maintained on time Historical Data Analysis Data about the flite. 17

17 Example :00 AM 7:00 AM 8:00 AM 9:00 AM 10:00 AM 11:00 AM 12:00 PM 1:00 PM 2:00 PM 3:00 PM 4:00 PM 5:00 PM 6:00 PM 7:00 PM 8:00 PM Ocean Aquarium Baseline Linear (Baseline) 18

18 Business Intelligence and Data Analytics Streams of data processed as data arrived to the processing engine. Data Processed in micro batches as soon as the batch filled with required amount of data. With small intervals. Data Processed on schedule in large batches. Ex. Daily, Weekly, Monthly. 19

Traditional Analytics Approaches Traditionally the majority of

around the concept of batch operations that move data between

19 Traditional Analytics Approaches Traditionally the majority of business intelligence and analytics solutions have been designed around the concept of batch operations that move data between different persisted data stores, such as relational databases or analytics cubes. Users would then issue a query against the data at rest to support scenarios such as ad-hoc analysis, dashboards or scorecards. 20

20 Workflow Data Stored in the source in a raw format Data stored in the Data Warehouse, Cubes and Data Marts Source DW, Cubes ETL Dashboards ETL Extracts data from the source, transforms the data and Bulk-loads data warehouse Data retrieved by user for analysis via User Interface (Dashboards, Reports) 21

21 Traditional Analytics Approaches 22

22 23

23 24

24 Microsoft Azure 25

25 Components The data connection to Stream Analytics is a data stream of events from a data source. This is called an "input." Stream Analytics has first-class integration with Azure data stream sources Event Hub, IoT Hub, and Blob storage that can be from the same or different Azure subscription as your analytics job. Azure Stream Analytics offers a SQL-like query language for performing transformations and computations over streams of events. Stream Analytics query language is a subset of standard T-SQL syntax for doing Streaming computations. In order to enable a variety of application patterns, Azure Stream Analytics has different options for storing output and viewing analysis results. This makes it easy to view job output and gives you flexibility in the consumption and storage of the job output for data warehousing and other purposes. 26

26 Main Components 27

27 Input Files

28 Azure Stream Analytics In cloud-to-device messages, send commands and notifications to your connected devices 03 Is a scalable publish-subscribe service that can ingest millions of events per second and stream them into multiple apps 02 Provides the flexibility and hyper-scale needed to store and retrieve large amounts of unstructured and media files 01 30

29 Azure Stream Analytics

30 Azure Stream Analytics

31 Azure Stream Analytics offers a SQL-like query language for performing transformations and computations over streams of events. SELECT * INTO [YourOutputAlias] FROM [YourInputAlias] 35

32 Overview

33 ASA Query Language Repeating, non-overlapping, fixed interval windows Generic window, overlapping, fixed size Slides by an epsilon and produces output at the occurrence of an event 37

Examples Unlike tumbling windows, hopping windows model scheduled overlapping

Note that a tumbling window is simply a hopping window whose hop is equal to

When using a sliding window, the system is asked to logically consider all

34 Examples Unlike tumbling windows, hopping windows model scheduled overlapping windows. Note that a tumbling window is simply a hopping window whose hop is equal to its size. When using a sliding window, the system is asked to logically consider all possible windows of a given length. Azure Stream Analytics instead outputs events only for those points in time when the content of the window actually changes, in other words when an event entered or exists the window. 38

35 39

Measure Consume. Store. Data Governance

Measure Consume. Store. Data Governance Collect Process Manage Measure Consume Store Data Governance Big Data Sources (Raw, Unstructured) Azure Machine Learning Business Insights Sensors Devices Intelligent Systems Service Hadoop on Windows