Business Intelligence in Azure Alex Whittles

Size: px

Start display at page:

Download "Business Intelligence in Azure Alex Whittles"

Anastasia Norton
6 years ago
Views:

1 Business Intelligence in Azure Alex Whittles PurpleFrogSystems.com

Alex Whittles SQL Relay Committee SQLRelay.co.uk SQL Bits Committee SQLBits.com Birmingham SQL UG SQLMidlands.com Birmingham Azure UG AzureBirmingham.

2 Alex Whittles SQL Relay Committee SQLRelay.co.uk SQL Bits Committee SQLBits.com Birmingham SQL UG SQLMidlands.com Birmingham Azure UG AzureBirmingham.uk MSc in Business Intelligence, CEng, CITP, FBCS, FIOEE, MIET, MIOD Run BI Consultancy Business Intelligence Consultancy Data Modelling Data Warehousing OLAP Cubes ETL Systems Reporting Systems Managed Service

3 Alex Whittles SQL Relay Committee SQLRelay.co.uk SQL Bits Committee SQLBits.com Birmingham SQL UG SQLMidlands.com Birmingham Azure UG AzureBirmingham.uk MSc in Business Intelligence, CEng, CITP, FBCS, FIOEE, MIET, MIOD Run BI Consultancy Business Intelligence Consultancy Data Modelling Data Warehousing OLAP Cubes ETL Systems Reporting Systems Managed Service

4 Cortana Analytics Suite Cortana Analytics Suite

5 Cortana Analytics Data Flow

6 Business Intelligence in Azure Options: 1. Migrating on-premise BI solutions to Azure 2. Batch load BI systems in Azure 3. Real-time BI systems in Azure For each: Staging / ETL / Data Warehouse / Cube

7 On-Premise Batch Load BI Source Data SQL Server Staging Database SQL Server Data Warehouse SSIS ETL SSAS Cube

8 Azure IAAS Solution [Infrastructure As A Service] On-Premise Azure Virtual Machine Source Data SSIS ETL SQL Server Staging Database SQL Server Data Warehouse SSIS ETL Don t do this! SSAS Cube

9 Better Azure IAAS Solution [Infrastructure As A Service] On-Premise Azure Virtual Machine Source Data VPN Gateway Or ExpressRoute SSIS ETL SQL Server Staging Database SQL Server Data Warehouse SSAS Cube

10 PAAS / IAAS Hybrid [Platform As A Service] On-Premise Azure Azure SQL DB Staging Database Azure SQL DB / SQL DW Data Warehouse Source Data VPN Gateway Or ExpressRoute SSIS ETL SSAS Cube Azure Virtual Machine

11 PAAS / IAAS Hybrid [Platform As A Service] On-Premise Azure Azure SQL DB Staging Database Azure SQL DB / SQL DW Data Warehouse Source Data VPN Gateway Or ExpressRoute SSIS ETL SSAS Cube (Tabular) Azure Virtual Machine

12 PAAS SSAS Tabular

13 PDW / APS / SQL DW PDW / APS = Parallel Data Warehouse / Analytics Platform System appliance, on premise SQL DW = Same concept but in Azure. Each node is an Azure SQL DB

14 Azure SQL Data Warehouse

15 Azure SQL DW Table Distribution Round Robin HASH

16 PAAS / IAAS Hybrid [Platform As A Service] On-Premise Azure Azure SQL DB Staging Database Azure SQL DB / SQL DW Data Warehouse Source Data VPN Gateway Or ExpressRoute SSIS ETL SSAS Cube (Tabular) Azure Virtual Machine

17 PAAS Solution [Platform As A Service] On-Premise Azure Source Data Azure Data Factory ELT Azure SQL DB Staging Database Azure SQL DB / SQL DW Data Warehouse PowerBI / SSAS Tabular

18 Example ELT Project CSV SQL Server Data Factory SQL DB etc. Landing SQL DW Staging Fact / Dim Stored Procs E L T

19 Cheaper Staging Blob/Table On-Premise Azure Source Data Azure Data Factory ELT Azure Blob Storage Staging Database Azure SQL DB / SQL DW Data Warehouse PowerBI / SSAS Tabular

20 Blob / Table Storage Blob Azure File System Any file, any format Unstructured Cheap Table Storage Simple tables No real indexing No foreign keys Basic data types Basic querying Structured Cheap

21 Integrate Data Lake On-Premise Azure Source Data Azure Data Factory ECLT Data Lake Landing SQL DB / SQL DW / DL Data Warehouse PowerBI / SSAS Tabular

22 Data Lake Store Data Lake Analytics Data Lake U-SQL (T-SQL + C#) Processing Engine Data Storage (HDFS) Structured Semi-Structured Unstructured

23 Example ECTL Project Excel CSV API SQL Server json E Data Factory C# Batch Compute U-SQL Data Lake Analytics Data Lake Landing Data Lake Clean Data Lake Out C T L Power BI

24 Example ECTLT Project Excel CSV API SQL Server json E Data Factory C# Batch Compute U-SQL Data Lake Analytics Data Lake Landing Data Lake Clean SQL DW Staging Stored Procs, SCD Handling C T L T SQL DW Fact / Dim

25 ETLT? Or ECLT? Or ECTLT?! Extract get the data Clean prepare the data files Transform create facts/dims? Load load into DW/etc. Transform create facts/dims? SCD handling Choose the right combination for your project

26 Data Lake Bridge between Normal & Big Data Data Lake Store Staging Data Lake Analytics Transform SQL DW/DB/DL Data Warehouse PowerBI / SSAS Data Lake Analytics SQL Queries / Reports Dashboards Excel Self-Service HD Insight Power User / Data Scientist big data analytics

27 Data Lake Analytics Vs HDInsight Data Lake U-SQL (T-SQL + C#) HDInsight Hadoop Map-Reduce Hive/Pig?

28 Real Time BI Real time Batch Data IoT or Event Hub Real-time data Data Lake Staging & Transform Stream Analytics Real-time insight SQL DW / DB / DL Data Warehouse PowerBI Data Lake Analytics Or HD Insight Power User / Data Scientist big data analytics SQL Queries / Reports Real-time Dashboards Excel Self-Service

29 Lambda Architecture Real time Batch Data IoT or Event Hub Real-time data Data Lake Staging & Transform Stream Analytics Real-time insight SQL DW / DB / DL Data Warehouse PowerBI Data Lake Analytics Or HD Insight Power User / Data Scientist big data analytics SQL Queries / Reports Real-time Dashboards Excel Self-Service

30 Machine Learning Real time Batch Data IoT or Event Hub Real-time data Data Lake Staging & Transform Stream Analytics Real-time insight SQL DW Data Warehouse PowerBI Machine Learning Data Lake Analytics Or HD Insight

31 Azure Machine Learning

32 Cortana Intelligence

33 Cortana Intelligence - Batch

34 Cortana Intelligence - RealTime

35 Cortana Intelligence Analytics

Azure Data Catalog Meta-data catalog Register New

for all Customer in my Azure Returns Columns & Data

SQL Server SQL DW SSAS SSRS Oracle Blob Table HDFS

36 Azure Data Catalog Meta-data catalog Register New Objects Descriptions Expert Tags Data Profile Search for all Customer in my Azure Returns Columns & Data Types Connection Strings Data Profile Etc. SQL Server SQL DW SSAS SSRS Oracle Blob Table HDFS Hive Teradata Data Lake MySQL HANA MDS Cosmos DB2 Salesforce Etc.

37 Register Object(s)

38 Search

39 U-SQL / Data Lake Demo

40 U-SQL string = EXTRACT dt string,ip string,address string,gender string,logtext string,fileyear int,filemonth int,file string USING Extractors.Text(delimiter:' '); * Schema on read

41 = SELECT dt.substring(5) AS dt,ip,((gender== "M" gender== "F")? gender : "") AS gender,logtext.substring(11).replace("]", "").Replace(":", "") AS logtext,myproject.udf.extracthousenumber( address") AS housenumber,row_number() OVER (PARTITION BY ip ORDER BY dt) AS rn WHERE fileyear == 2017; TO "/out/mydata.csv" USING Outputters.Text(quoting : true, outputheader : true);

42 U-SQL User Defined Functions

43 U-SQL / Data Lake

44 U-SQL / Data Lake

45 Data Lake Store Data Lake Analytics Data Lake U-SQL (T-SQL + C#) Processing Engine Data Storage (HDFS) Structured Semi-Structured Unstructured

46 In Summary Virtual Machine with On-Prem solution? Or genuine Azure solution? Real-time, Batch, or both? Lambda Architecture U-SQL & Data Lake <- Everything to Everyone! E C T L T? Missing: SSIS Data flow equivalent

Business Intelligence in Azure Alex@PurpleFrogSystems.

47 Business Intelligence in

Azure ML Data Camp. Ivan Kosyakov MTC Architect, Ph.D. Microsoft Technology Centers Microsoft Technology Centers. Experience the Microsoft Cloud

Azure ML Data Camp. Ivan Kosyakov MTC Architect, Ph.D. Microsoft Technology Centers Microsoft Technology Centers. Experience the Microsoft Cloud Microsoft Technology Centers Microsoft Technology Centers Experience the Microsoft Cloud Experience the Microsoft Cloud ML Data Camp Ivan Kosyakov MTC Architect, Ph.D. Top Manager IT Analyst Big Data Strategic