Big Data & Advanced Analytics - "managed Services on Azure

Size: px

Start display at page:

Download "Big Data & Advanced Analytics - "managed Services on Azure"

Dorcas Griffin
6 years ago
Views:

1 Big Data & Advanced Analytics - "managed Services on Azure Guido Jacobs Global Black Belt TSP Big Data Guido.Jacobs@microsoft.com Microsoft Deutschland GmbH

East Quebec City East US Virginia East US 2 Virginia US DoD East TBD United Kingdom Regions North Europe Ireland West Europe Netherlands Germany North East ** Magdeburg Germany Central **

2 Hyper scale Infrastruktur das ist Azure! US DoD West TBD West US California US Gov Iowa West US 2 California West Central US Central US Iowa North Central US Illinois South Central US Texas Canada Central Toronto US Gov Virginia Canada East Quebec City East US Virginia East US 2 Virginia US DoD East TBD United Kingdom Regions North Europe Ireland West Europe Netherlands Germany North East ** Magdeburg Germany Central ** Frankfurt India West Mumbai India Central Pune SE Asia Singapore China South * Shanghai India South Chennai China North * Beijing East Asia Hong Kong Korea Central Seoul Japan East Tokyo, Saitama Japan West Osaka Korea South Australia East New South Wales Brazil South Sao Paulo State Australia South East Victoria

3 Hadoop im traditionellen Sinne N1 N2

4 Hadoop im traditionellen Sinne N1 N2 N1 N1 N1 N3 N2 N2 N2 N4 N3 N3 N5 N4 N4 N6 N7 N8

5 BIG DATA STORAGE BIG DATA ANALYTICS User Adoption Flexibilität in der Cloud CONTROL EASE OF USE Azure Data Lake Analytics Azure HDInsight Azure Marketplace HDP CDH MapR Any Hadoop technology Workload optimized, managed clusters Specific apps in a multitenant form factor IaaS Hadoop Managed Hadoop Big Data as-a-service Azure Data Lake Analytics Azure Data Lake Store Azure Storage

6 Azure Data Lake Store A hyper-scale repository for Big Data analytics workloads Hadoop File System (HDFS) for the cloud No limits to scale Store any data in its native format Enterprise-grade access control, encryption at rest Optimized for analytic workload performance

7 Azure HDInsight Hadoop and Spark as a Service on Azure Fully-managed Hadoop and Spark for the cloud 100% Open Source Hortonworks data platform Clusters up and running in minutes Managed, monitored and supported by Microsoft with the industry s best SLA Familiar BI tools for analysis, or open source notebooks for interactive data science 63% lower TCO than deploy your own Hadoop on-premises* *IDC study The Business Value and TCO Advantage of Apache Hadoop in the Cloud with Microsoft Azure HDInsight

8 Azure Data Lake Analytics A new distributed analytics service Distributed analytics service built on Apache YARN Elastic scale per query lets users focus on business goals not configuring hardware Includes U-SQL a language that unifies the benefits of SQL with the expressive power of C# Integrates with Visual Studio to develop, debug, and tune code faster Federated query across Azure data sources Enterprise-grade role based access control

Daten einfach kombinieren Azure Data Lake Storage Vorteile Azure Storage Blobs Große Datenmengen müssen NICHT zwischen unterschiedlichen

weniger Kopien Eine Abfragesprache für ALLE Datenquellen Jeder Datenspeicher behält seine Souveränität Bedarfsbezogenen Lösungsdesign SQL

9 Daten einfach kombinieren Azure Data Lake Storage Vorteile Azure Storage Blobs Große Datenmengen müssen NICHT zwischen unterschiedlichen Speichern verschoben werden Einheitliche Sicht auf Daten unabhängig vom physikalischen Speicherplatz Verringerung der Datenpflege durch weniger Kopien Eine Abfragesprache für ALLE Datenquellen Jeder Datenspeicher behält seine Souveränität Bedarfsbezogenen Lösungsdesign SQL Prädikate werden an die SQL-Quellen gesendet Filters Joins U-SQL Query Azure Data Lake Analytics Query SQL in Azure VMs Azure SQL DB Azure SQL Data Warehouse

10 Trennung von Storage & Compute ADL-A/HDI Data Azure Data Lake Store SQL DB PowerBI ADL-A/HDI

11 Trennung von Storage & Compute (2) ADL-A/HDI HDI/ R-Server HDI/ H2O AI Data Azure Data Lake Store SQL DW PowerBI ADL-A/HDI ADL-A/HDI ADL-A/HDI

12 What is Azure Data Catalog? An enterprise-wide catalog directory in Azure in Azure that enables that enables self-service selfservice discovery of data from any source discovery of data from any source A metadata repository that allow users to register, enrich, understand, discover, and consume data sources

Enterprise grade Security in HDInsight Authentication Kerberos Azure Active Directory Perimeter Level Security Virtual Networks Network Security

13 Enterprise grade Security in HDInsight Authentication Kerberos Azure Active Directory Perimeter Level Security Virtual Networks Network Security Groups (Firewalls) Authorization Apache Ranger RBAC for Admin POSIX ACLs for Data Plane Data Security Server-Side encryption at rest HTTPS/ TLS in-transit

14 HDInsight - 3rd Party Solutions H2O Sparkling Water: Datameer: Data iku - Data Science Studio: Cask Data App Platform (CDAP): StreamSets Data Collector: Spark Job Server for KNIME Spark Executor: SnapLogic Hadooplex:

15 Shared Meta-Data HDInsight Cluster Type: Spark HN HN HDInsight Cluster Type: 3 rd Party HN HN 3 rd Party Components Azure Big Data Storage/ Azure Data Lake Store HIVE OOZIE RANGER WN WN WN RAM & CPU are configured to fulfill the workload requirements HDInsight Cluster Type: R-Server HN HN Edge-Node WN WN WN WN WN WN RAM & CPU are configured to fulfill the workload requirements HDP (IaaS) Type: Cloudbreak HN HN HN WN WN WN Optimized for MMP based Analytical Workloads Authorized Access by Azure AD Access via: ADL:// (Oauth2) WebHDFS (Oauth2) RAM & CPU are configured to fulfill the workload requirements HDFS (only for temp & spilling data) No upfront cost No pre-allocation Pay-for-stored-Data On-premise HDP Cluster Synchronisation on File-level

RESOURCES ACROSS THE SALES CYCLE The Business Value and TCO of HDInsight 418% 5-year ROI Four month payback period 63% 5-year lower TCO than on-premises 66% staff

18 RESOURCES ACROSS THE SALES CYCLE The Business Value and TCO of HDInsight 418% 5-year ROI Four month payback period 63% 5-year lower TCO than on-premises 66% staff efficiencies than on-premises Get it at *IDC study The Business Value and TCO Advantage of Apache Hadoop in the Cloud with Microsoft Azure HDInsight

Insights to HDInsight

Insights to HDInsight Why Hadoop in the Cloud? No hardware costs Unlimited Scale Pay for What You Need Deployed in minutes Azure HDInsight Big Data made easy Enterprise Ready Easier and more productive