Apache Hadoop in the Datacenter and Cloud

Size: px

Start display at page:

Download "Apache Hadoop in the Datacenter and Cloud"

Barnard Jacobs
5 years ago
Views:

1 Apache Hadoop in the Datacenter and Cloud

2 The Shift to the Connected Data Architecture Digital Transformation fueled by Big Data Analytics and IoT ACTIONABLE INTELLIGENCE Cloud and Data Center IDMS Relational Database Data in Motion Data at Rest Powered by Open Source System centric Mainframe Client / Server Web and SaaS 2 Hortonworks Inc All Rights Reserved Modern Applications Connected Data Architecture User centric Transformational Use Cases Predictive Retail Factory Automation Connected Cars Predictive Analytics Artificial Intelligence

3 Hadoop in the Data Center Create and Manage Central Data Lakes Support all Types of Data Provide Flexible Processing and Access Methods Reduce Architecture Costs by 80% or More Drive Transformational New Use Cases 3 Hortonworks Inc All Rights Reserved

4 Hadoop in the Cloud Fast On Ramp for New Users Elastic Compute and Storage Capabilities Zero configuration access engine capabilities (HD Insight) Eliminate Hardware purchases Facilitate Certain Modern Data Applications through Cloud Connectivity 4 Hortonworks Inc All Rights Reserved

5 Transformational Applications Require Connected Data Edge Analytics Machine Learning CLOUD Edge Data Data in Motion Data at Rest Stream Analytics DATA CENTER Data in Motion Data at Rest Edge Data Deep Historical Analysis Hortonworks Inc All Rights Reserved

6 Our Focus: Enable Modern Applications on Connected Data Platforms Continuous Insights Enterprise Ready Any Delivery Model Open Innovation Deliver insights from ALL data, origin to rest Management Security Governance Data Center Cloud Hybrid Architecture Community Ecosystem Hortonworks Inc All Rights Reserved

Actionable Intelligence from Connected Data Platforms Modern Data Applications Capturing perishable insights from data in motion Ensuring rich, historical insights on data at rest

8 Actionable Intelligence from Connected Data Platforms Modern Data Applications Capturing perishable insights from data in motion Ensuring rich, historical insights on data at rest Necessary for modern data applications DATA IN MOTION ACTIONABLE INTELLIGENCE DATA AT REST Hortonworks DataFlow Hortonworks Data Platform 8 Hortonworks Inc All Rights Reserved

10 Hortonworks Data Platform 2.5 Highlights Dynamic Security: Apache Atlas + Ranger Integration Enterprise Spark at Scale: Apache Zeppelin Notebook for Spark Real Time Applications: Storm and HBase/Phoenix Streamlined Operations: Apache Ambari Interactive Query in Seconds: Hive with LLAP (Technical Preview ) 10 Hortonworks Inc All Rights Reserved

12 Introducing Tag Based Security Apache Atlas and Ranger Integration Basic Tag policy Access and entitlements can be based on attributes. As an example: Personally Identifiable Information (PII) is a tag that can be leveraged to protect sensitive personal data. Geo based policy Access policy based on location. As an example: A user might be able to access data in North America, but may be restricted from access in EMEA due to privacy compliance. Time based policy Access policy based on time windows. An an example: A user might be able to access data only between 8AM 5PM (common in SOX regulations.) Prohibitions Restrictions on combining two data sets which might be in compliance originally, but not when combined together. As an example, SSNs and Names) Key Benefits: New scalable metadata based security paradigm Dynamic, real time policy Automatic updates to changes in metadata Centralized and simple to manage policy 12 Hortonworks Inc All Rights Reserved

13 Apache Atlas Powers Cross Component Data Lineage As a part of HDP 2.5, users can track lineage across the following components using Atlas: Apache Sqoop Import from and export to relational databases, and additional package that leverages Sqoop Hive Dataset lineage with entity versioning (including schema changes) Apache Kafka/ Storm IoTevent level processing, such as syslogs or sensor data Falcon Data lifecycle at Feed and Process entity level for replication, and repeating workflows. Tracks period icy, throttling, eviction. ATLAS 69 FALCON 1570 Key Benefits: Enterprises need open solutions, not single app vendor More native connectors than any other vendor Hardened metadata infrastructure 13 Hortonworks Inc All Rights Reserved

Expanded Native Connector: Dataset Lineage

Custom Activity Reporter Metadata Repository

15 Apache Atlas Enables Business Catalog for Ease of Use Organize data assets along business terms Authoritative: Hierarchical business Taxonomy Creation Agile modeling: Model Conceptual, Logical, Physical assets Definition and assignment of tags like PII (Personally Identifiable Information) Comprehensive features for compliance Multiple user profiles including Data Steward and Business Analysts Object auditing to track Who did it Metadata Versioning to track what did they do Faster Insight: Data Quality tab for profiling and sampling User Comments Key Benefits: Easy way to create business Taxonomy Useful for multiple user types including Data Steward and Business Analysts Comprehensive features for compliance 15 Hortonworks Inc All Rights Reserved

17 Streamlining Operations, Three Phase Plan Focused Strategic Investments into our core products to give customers more unique tooling to quickly understand the cluster s health, how business users are using it, and where to focus efforts when issues arise. Capabilities Phase 1: Advanced Performance & Health Metrics Dashboards with Ambari Phase 2: Consolidated Cluster Activity Reporting NEW! with SmartSense Phase 3: Centralized & Contextual Log Search Tech Preview with Ambari Core Technologies Apache Ambari Ambari Metrics System Apache Solr Hortonworks SmartSense Grafana Ambari Metrics System Grafana Solr AMBARI Log Search Dedicated UIs SmartSense 17 Hortonworks Inc All Rights Reserved

Streamlined Operations Phase 1: Advanced Metrics Visualization & Dashboarding Grafana Goal: Quickly understand cluster health metrics and key performance indicators Ambari Metrics System AMBARI

18 Streamlined Operations Phase 1: Advanced Metrics Visualization & Dashboarding Grafana Goal: Quickly understand cluster health metrics and key performance indicators Ambari Metrics System AMBARI Capabilities Centralized Dashboarding focusing on component Health & Performance Ad Hoc Graph Creation Pre Built Dashboards HDFS YARN HBase Core Technologies Ambari Metrics System Grafana 18 Hortonworks Inc All Rights Reserved

Streamlined Operations Phase 2: Consolidated Cluster Activity Reporting AMBARI Ambari Ambari Metrics Metrics System System SmartSense Apache Zeppelin Goal: Quickly visualize and report on how

20 Streamlined Operations Phase 2: Consolidated Cluster Activity Reporting AMBARI Ambari Ambari Metrics Metrics System System SmartSense Apache Zeppelin Goal: Quickly visualize and report on how business users and tenants are using the cluster, top 10 queues, users, most time consuming jobs Capabilities Top K Activity Reporting Chargeback Services Covered YARN MapReduce Hive/Tez Spark HDFS 20 Hortonworks Inc All Rights Reserved Core Technologies Hortonworks SmartSense Apache Zeppelin

Preview: Streamlined Operations Investments Phase 3: Centralized & Contextual Log Search AMBARI Goal: When issues arise, be able to quickly find issues across all HDP components Solr Log Search

22 Preview: Streamlined Operations Investments Phase 3: Centralized & Contextual Log Search AMBARI Goal: When issues arise, be able to quickly find issues across all HDP components Solr Log Search Capabilities Rapid Search of all HDP component logs Search across time ranges, log levels, and for keywords Core Technologies: Apache Ambari Apache Solr Apache Ambari Log Search 22 Hortonworks Inc All Rights Reserved

25 Hive 2 with LLAP Enable Interactive Query In Seconds Developer Productivity: Interactive query in seconds Ease of Use and Adoption : 100% compatible with Hive SQL Enterprise Readiness: Linear scaling at Terabytes volume of data Streamlined Operations: LLAP integration with Ambari with automated dashboards 25 Hortonworks Inc All Rights Reserved

Hive 2 with LLAP: Preliminary Numbers 80 Hive2.0 and LLAP: TPC DS at 10 TB Scale, 18 Nodes 70 60 Min query time: Query 55: 2.38s 50 40 Hive2.

26 Hive 2 with LLAP: Preliminary Numbers 80 Hive2.0 and LLAP: TPC DS at 10 TB Scale, 18 Nodes Min query time: Query 55: 2.38s Hive2.0 Tez LLAP q3 q7 q12 q13 q19 q21 q26 q27 q42 q43 q45 q52 q55 q60 q73 q84 q89 q91 q98 26 Hortonworks Inc All Rights Reserved

Data Center. Always on enterprise use cases are common. Elasticity, Automation, Pay as you Go, One Click Start. Ephemeral use cases are common starting point.

30 How Do We Approach The Cloud Market? HYBRID SEGMENT Today s enterprise customers CLOUD ONRAMP New users via digital engagement or existing customers exploring cloud options Seamless Connected Data Architecture across Cloud and Data Center. Always on enterprise use cases are common. Elasticity, Automation, Pay as you Go, One Click Start. Ephemeral use cases are common starting point. AzureHDInsight, HDP, and HDF are our Premier offerings. Customer journey to future state architecture, cloud operation & consumption model. AzureHDInsightis our Premier offering. Focused offerings for AWS that enable us to engage and position our Premier offerings. Cloud first approach to product design, development, testing & delivery 30 Hortonworks Inc All Rights Reserved

31 Outlook: Cloud and the Big Data Market Public cloud adoption (AWS, Azure, Google) will continue to accelerate Many customers will go Cloud First to simplify/speed adoption Customers deploying in public cloud expect a pay as you go (PAYG) pricing model Hourly pricing is default; reserved optimizes annual spend; spot optimizes hourly spend Interested in running workloads in the cloud and in addition to on premiseclusters. Familiar with Native Cloud tooling. Heightens importance of product packaging and user experience tuned to Cloud 31 Hortonworks Inc All Rights Reserved

Microsoft Azure HDInsights Powered by Hortonworks Data Platform Seamless Access to the Public Cloud for Spark, Hive, and HBaseand other mission critical workloads Unmatched Economics combining

33 Microsoft Azure HDInsights Powered by Hortonworks Data Platform Seamless Access to the Public Cloud for Spark, Hive, and HBaseand other mission critical workloads Unmatched Economics combining HDInsight selasticity in the cloud with HDP s cost efficiencies at scale Enterprise Readiness with robust security, governance and operations in the cloud, powered by Hortonworks Data Platform 33 Hortonworks Inc All Rights Reserved

Connected Data Architecture with Azure HDInsight CLOUD Azure HDInsight Cloud Data Processing HDInsightCluster Types Ideal Use Cases Data Prep, Query, and Analysis (Hadoop, Hive, Pig) Iterative In

34 Connected Data Architecture with Azure HDInsight CLOUD Azure HDInsight Cloud Data Processing HDInsightCluster Types Ideal Use Cases Data Prep, Query, and Analysis (Hadoop, Hive, Pig) Iterative In Memory Analysis (Spark) HDF Data Flow Management Advanced Statistics, Modeling, Machine Learning (R Server on Spark) NoSQLData Storage (HBase) DATA CENTER HDP Enterprise Data Lake Real time Event Processing (Storm) Hortonworks Inc All Rights Reserved

Runs in more datacenters than anyone else Central US Iowa North Central US Illinois West

Virginia East US 2 Virginia North Europe Ireland India Central Pune China South * Shanghai

East New South Wales Brazil South Sao Paulo State Australia South East Victoria Azure

35 Runs in more datacenters than anyone else Central US Iowa North Central US Illinois West Europe Netherlands China North * Beijing West US California South Central US Texas East US Virginia East US 2 Virginia North Europe Ireland India Central Pune China South * Shanghai Japan East Tokyo, Saitama Japan West Osaka East Asia Hong Kong SE Asia Singapore Australia East New South Wales Brazil South Sao Paulo State Australia South East Victoria Azure doubling compute and storage every 6 months 35 Hortonworks Inc All Rights Reserved

36 Microsoft Azure HDInsight and Apache Projects in the Cloud YARN DATA OPERATING SYSTEM Batch STORAGE GOVERNANCE OPERATIONS SECURITY STORAGE Machine Learning Standard Hadoop Projects for Hive, YARN, HDFS, MapReduce, Pig, Tez, Sqoop, oozie, Zookeeper, Mahout, Phoenix CompehensiveList of Emerging Projects Spark, Storm Hbase, and R Interactive Streaming Ability to Add Projects Add various projects to the the cloud Search 36 Hortonworks Inc All Rights Reserved

Forrester Wave : Big Data HadoopCloud Solutions,

Compel Enterprise Adoption Of Hadoop In The Cloud

Connected Data Architecture with HDC for AWS CLOUD HDF Data Flow Management HDC for AWS Cloud Data Processing Ideal Use Cases Data Science and Exploration (Spark, Zeppelin) ETL

38 Connected Data Architecture with HDC for AWS CLOUD HDF Data Flow Management HDC for AWS Cloud Data Processing Ideal Use Cases Data Science and Exploration (Spark, Zeppelin) ETL and Data Preparation (Hive, Spark) DATA CENTER Hortonworks Inc All Rights Reserved HDP Enterprise Data Lake TECH PREVIEW Analytics and Reporting (Hive2 w/llap, Zeppelin)

41 Why Hortonworks Cloud Solutions? Choice of Cloud Rich Set of Capabilities and Security Zero configuration access engine capabilities (HD Insight) S3 Integrations on AWS (Tech Preview) Award Winning Hadoop Expertise 41 Hortonworks Inc All Rights Reserved

42 Connected Data Platforms Integrate Cloud and Data Center Deployments Edge Analytics Machine Learning CLOUD Edge Data Data in Motion Data at Rest Stream Analytics DATA CENTER Data in Motion Data at Rest Edge Data Deep Historical Analysis Hortonworks Inc All Rights Reserved

Insights to HDInsight

Insights to HDInsight Why Hadoop in the Cloud? No hardware costs Unlimited Scale Pay for What You Need Deployed in minutes Azure HDInsight Big Data made easy Enterprise Ready Easier and more productive