Azure Data Lake How to organize. Jan Cordtz, Microsoft Denmark Cloud Solution Architect

Size: px

Start display at page:

Download "Azure Data Lake How to organize. Jan Cordtz, Microsoft Denmark Cloud Solution Architect"

Barry Scott
5 years ago
Views:

1 Azure Data Lake How to organize Jan Cordtz, Microsoft Denmark Cloud Solution Architect

Platform as a Service Security & Management Security Center Portal Azure Active Directory Azure AD B2C Multi-Factor Authentication Media Services API Management Logic Apps Media Analytics BizTalk

Cache Storage Tables DocumentDB Azure Search Cognitive Services Bot Framework Cortana Hybrid Cloud Azure AD Health Monitoring AD Privileged Identity Management Domain Services Backup Automation

2 Platform as a Service Security & Management Security Center Portal Azure Active Directory Azure AD B2C Multi-Factor Authentication Media Services API Management Logic Apps Media Analytics BizTalk Services Service Bus Content Delivery Network Web Apps API Apps Service Fabric Mobile Apps Cloud Services Notification Hubs Functions SQL Database SQL Server Stretch Database SQL Data Warehouse Redis Cache Storage Tables DocumentDB Azure Search Cognitive Services Bot Framework Cortana Hybrid Cloud Azure AD Health Monitoring AD Privileged Identity Management Domain Services Backup Automation Operational Analytics Scheduler Key Vault Store/ Marketplace VM Image Gallery & VM Depot Container Service Batch VM Scale Sets RemoteApp Visual Studio VS Team Services Application Insights Mobile Engagement Xamarin HockeyApp HDInsight/ Databricks IoT Hub Data Catalog Event Hubs Machine Learning Data Lake Analytics Service Data Factory Stream Analytics Data Lake Store Power BI Embedded Import/Export Azure Site Recovery StorSimple Infrastructure as a Service Data Center

REGIONAL INDUSTRY GLOBAL More certifications than any cloud provider ISO 27001 ISO 27018 ISO 27017 ISO

PCI DSS Level 1 CDSA MPAA FACT UK Shared Assessments FISC Japan HIPAA / HITECH Act Trusted HITRUST GxP 21

China GB 18030 China TRUCS Singapore MTCS Australia IRAP/CCSL New Zealand GCIO Japan My Number Act ENISA

3 REGIONAL INDUSTRY GLOBAL More certifications than any cloud provider ISO ISO ISO ISO SOC 1 Type 2 SOC 2 Type 2 SOC 3 CSA STAR Self-Assessment CSA STAR Certification CSA STAR Attestation PCI DSS Level 1 CDSA MPAA FACT UK Shared Assessments FISC Japan HIPAA / HITECH Act Trusted HITRUST GxP 21 CFR Part 11 MARS-E IG Toolkit UK FERPA GLBA FFIEC Argentina PDPA EU Model Clauses UK G-Cloud China DJCP China GB China TRUCS Singapore MTCS Australia IRAP/CCSL New Zealand GCIO Japan My Number Act ENISA IAF Japan CS Mark Gold Spain ENS Spain DPA India MeitY Canada Privacy Laws Privacy Shield Germany IT Grundschutz workbook

4 Data Analytics Cloud Always been there but growing rapidly Been their for a long time (BI) but getting much more advanced Machine Learning/AI New kid on the block Unlimited compute/storage Fast deployment Pay-as-you go Many services

5 Open and hybrid

7 Business needs - Governance - Organize - Common understanding of data - Trial: Error/Proceed - Hot/Cold path - No specific technology - Flexible economy - Hybrid Mode 1 - Datawarehouse - Reporting Central platform Data Lake / Data Bank Mode 2 - IOT - Machine Learning - Analytics Selfservice - Dashboard - Business Intelligence

8 Built on Open Standards Built on YARN Analytics Service U-SQL HDInsight d Store lets all HDFS compliant analytic applications connect to it like Hortonworks, Cloudera, and MapR Microsoft HDInsight is 100% Apache Hadoop YARN HDFS Microsoft continues to contribute tens of thousands of code and engineering hours to open source Store

9 A databank Fixed Sizeable IAAS Economics Dynamic Sizeable PAAS Economics Cosmos DB Operational System A SQL DW Operational System B Data Ingestion Data Lake Cube Operational System X Machine Learning DataMart Archive storage Data Ingestion Data Bricks External Data Whatever Apps

10 Data storage Azure Machine Learning services Data Science Virtual Machine SQL DB SQL DW Machine Learning Studio DataBricks Microsoft SQL (General) PostgresSQL (GIS) Key-Value (Table) Column-Value (Cassandra) Documents (MongoDB) Cloudera* HDInsight MySQL (LAMP/PHP) Graph (Tinkerpop) Rest Files HortonWorks* MapR* SQL NoSQL Blob Data Lake DB Cube

11 Storage from a functionality point of view Functionality and cost ETL File storage Data Lake ETL Data Factory Data Factory Database Cube 11

12 Principal regarding the Organization Is very simple to use for an end-user/application (=flat file/csv file) Is as cost-effective as sensible/possible. Do not compromise security. Fits well into a DevOps scenario Automatic meta-tagging Have a well-defined path for the information needed to be able to support an effective auditing and logging process.

13 Organizing the Azure Data Lake Azure Data Lake Publish A Folder per area Read/Write Users in Groups Read Only except Work System Account(s) Data Ingestion Copy Work Landing Zone Landing Zone System Account(s) read/write Work System Account(s) - read Work System Account(s) read/write Transform & Anonymize Transform Publish B Users in Groups Publish X Users in Groups All data Folder per area Read/Write Read Only except Work System Account(s) Folder per area Read/Write Read Only except Work System Account(s) Analytics Read/Write Archive Users in Groups Read Only except Work System Account(s) Data Catalog

14 Data Ingestion Push/ Pull Hot/Cold Path Standardization Items like : Date formats (yyyymmdd), number formats (,. or.,) Validation Is the content you are coming with in accordance with what we have agreed Gatekeeper Are you allowed to enter? Examples SSIS, Event Hub, Data Factory. Database, FTP, File Storage. Firewall, AD control.

15 Azure Data Lake and DevOps Azure Data Lake Publish A Folder per area Read/Write Work Users in Groups Publish B Read Only except Work System Account(s) Folder per area Read/Write Data Ingestion Copy Landing Zone Anonymize Transform Transform Users in Groups Publish X Users in Groups Read Only except Work System Account(s) Folder per area Read/Write Read Only except Work System Account(s) Analytics All data Read/Write Users in Groups Read Only except Work System Account(s) Data Catalog

16 Thank you

Data Lake Organization A Hadoop Eco-System. Jan Cordtz, Microsoft Denmark Cloud Solution Architect

Data Lake Organization A Hadoop Eco-System Jan Cordtz, Microsoft Denmark jcordtz@microsoft.com Cloud Solution Architect Hyper scale Infrastructure 100+ Datacenters across 42 Regions Worldwide Learn more: