Managing Enterprise Hadoop Clusters with Apache Ambari. Jayush Hortonworks Apache Ambari PMC

Size: px

Start display at page:

Download "Managing Enterprise Hadoop Clusters with Apache Ambari. Jayush Hortonworks Apache Ambari PMC"

Tobias Freeman
6 years ago
Views:

Apache Ambari Themes Operate Hadoop at Scale Deliver

integraqon with exisqng enterprise systems, such as

Extend for the Ecosystem Provide extensible plaiorm

5 Apache Ambari Themes Operate Hadoop at Scale Deliver the core operaqonal capabiliqes to provision, manage and monitor Hadoop clusters at scale. Integrate with the Enterprise Robust API for integraqon with exisqng enterprise systems, such as MicrosoE SCOM and Teradata Viewpoint. Extend for the Ecosystem Provide extensible plaiorm for Customers, Partners and the Community (Stacks, Views) 5 Hortonworks Inc All Rights Reserved

9 Fast forward 5 years to today Ã Latest JIRA: AMBARI Ã 150+ Contributors Ã 60+ CommiKers Ã JIRAs filed Ã JIRAs fixed At 1.5 day per JIRA ~ 90 person years! Ã Used by hundreds of companies 9 Hortonworks Inc All Rights Reserved

10 Ambari 3rd Biggest Project Apache #2: Hadoop at ~32k as it is split across mulqple JIRA Projects #1 #3 #4 #5 * Based on total JIRAs filed on a project basis as of April 26, Hortonworks Inc All Rights Reserved

11 Timeline Ambari Stacks Ambari Blueprints Ambari Views Alerts Framework Metrics System Rolling Upgrade Kerberos AutomaQon Enhanced Dashboards Smart Configs Express Upgrade Ambari 1.5.* Apr 2014 Ambari 1.6.* May 2014 Ambari 1.7.* Dec 2014 Ambari 2.0.* April 2015 Ambari 2.1.* July 2015 Ambari 2.2.* Dec JIRAs 908 JIRAs 1620 JIRAs 1804 JIRAs 2674 JIRAs 1542 JIRAs Current GA Version (2.2.2) 11 Hortonworks Inc All Rights Reserved ResoluQon of 9k+ JIRAs

Agenda Ambari Overview Ambari Features Demo Q&A

To share best pracqces on layout and cluster configuraqon Views To extend and customize the Ambari Web

13 Extensibility Features Stacks To add new Services (ISV or otherwise) beyond HDP stack To customize a Stack for customer specific environments Blueprints To use Ambari for automaqng cluster installaqons. To share best pracqces on layout and cluster configuraqon Views To extend and customize the Ambari Web UI Add new capabiliqes, customize exisqng capabiliqes 13 Hortonworks Inc All Rights Reserved

16 Stack Terminology Term DefiniKon Examples STACK Defines a set of Services, where to obtain the sooware packages and how to manage the lifecycle. HDP- 2.3, HDP- 2.2 SERVICE Defines the Components that make- up the service. HDFS, NAGIOS, YARN COMPONENT The building- blocks of a Service, that adhere to a certain lifecycle. NAMENODE, DATANODE, OOZIE_SERVER CATEGORY The category of Component. MASTER, SLAVE, CLIENT REPO Repository metadata where the arqfacts reside hkp://public- repo- 1.hortonworks.com/HDP/ centos6/2.x/ga/ Hortonworks Inc All Rights Reserved

Ambari Stack Ã Stacks define Services + Repo What is a stack, and where to

Service Ã Each service has defined lifecycle commands start, stop, status,

define custom commands Ambari Server Stack Service DefiniQons xml Repos

17 Ambari Stack Ã Stacks define Services + Repo What is a stack, and where to get the bits Ã Each service has a definiqon What components are part of the Service Ã Each service has defined lifecycle commands start, stop, status, install, configure Ã Lifecycle is controlled via command scripts Ã Ability to define custom commands Ambari Server Stack Service DefiniQons xml Repos Command Scripts python Ambari Agents 17 Hortonworks Inc All Rights Reserved

18 Stacks Support Inheritance HDP 2.1 Stack Overrides any Service definiqons, commands and configuraqons Adds new Services specific to this Stack HDP 2.0 Stack Defines a set of Service definiqons Default service configuraqons and command scripts 18 Hortonworks Inc All Rights Reserved

20 Automated Cluster Deployment Ã Ã Deploy clusters of any scale with ease Two REST API calls is all it takes to provision a cluster Who uses it? Ã Ã HDInsight (Microsoo Azure) Hortonworks QA 20 Hortonworks Inc All Rights Reserved

21 Example: Create a 100- node Cluster 1. POST /api/v1/blueprints/my-blueprint! 2. POST /api/v1/clusters/my-cluster! {! "configurations" : [! {! hdfs-site" : {!! "dfs.datanode.data.dir" : /hadoop/1,/hadoop/2,/hadoop/3"! }! }! ],! "host_groups" : [! {! "name" : master-host",! "components" : [! { "name" : "NAMENODE },! { "name" : "RESOURCEMANAGER },!! ],! "cardinality" : "1"! },! {! "name" : worker-host",! "components" : [! { "name" : DATANODE },! { "name" : NODEMANAGER },!! ],! "cardinality" : "1+"! },! ],! "Blueprints" : {! "stack_name" : "HDP",! "stack_version" : "2.0"! }! }! {! "blueprint" : my-blueprint",! "host_groups" :[! {! "name" : master-host",! "hosts" : [! {! "fqdn" : master001.ambari.apache.org!! }! ]! },! {! "name" : worker-host",! "hosts" : [! {! "fqdn" : worker001.ambari.apache.org!! },! {! "fqdn" : worker002.ambari.apache.org!! },!! {! "fqdn" : worker099.ambari.apache.org!! }! ]! }! ]! }! 21 Hortonworks Inc All Rights Reserved

22 Cluster ReplicaKon Ã Ã Export blueprint from an exisqng cluster Import blueprint to replicate the cluster GET/api/v1/clusters/my-cluster? format=blueprint! {! "configurations" : [! {! cluster-env" : {!! user_group" : hadoop"! }! hdfs-site" : {!! "dfs.datanode.data.dir" : /hadoop/1,/hadoop/2,/hadoop/3"! }! }! ],! "host_groups" : [! {! "name" : master-host",! "components" : [! { "name" : "NAMENODE },! { "name" : "RESOURCEMANAGER },!! ],! "cardinality" : "1"! }! ],! "Blueprints" : {! "stack_name" : "HDP",! "stack_version" : "2.0"! }! }! 22 Hortonworks Inc All Rights Reserved

23 Blueprint Features Ambari 2.0: Ã Ã High availability (HA) cluster deployments Adding hosts using blueprints (AMBARI- 8458) Ambari 2.1: Ã Advanced cluster creaqon opqons (AMBARI ) Ambari 2.2: Ã Ã Kerberized cluster deployments (AMBARI ) Stack advisor recommendaqons (AMBARI ) 23 Hortonworks Inc All Rights Reserved

25 Stack Upgrades Rolling vs Express Upgrade modes Side- by- Side Bits and Configs Bits: /usr/hdp/ /usr/hdp/ /usr/hdp/ Configs: /etc/hive/conf/ (iniqal) /etc/hive/conf/v0 (HDP ) /etc/hive/conf/v1 (HDP 2.3) minor jump major jump Hortonworks Inc All Rights Reserved

26 Express vs Rolling Upgrade Rolling Upgrade Ã Ã Ã Ã Services are up the enqre Qme Upgrade one component at a Qme Robust and fault- tolerant Service checks performed frequently during the upgrade Express Upgrade Ã Ã Ã Ã All services are brought down, upgraded and restarted Faster upgrade mode Planned service downqme RelaQvely service checks performed less frequently during the upgrade. 26 Hortonworks Inc All Rights Reserved

Stack Upgrade Install Version Install new version in parallel on all

29 Stack Upgrade Upgrade Catalog Ã Upgrades are driven by upgrade catalogs defined in stack definiqons. Ã Defines upgrade groups and upgrade order Ã Provides ability to modify configuraqons Set, move, delete, transform Ã Upgrade steps can be marked as skippable and retryable Ã Supports execuqng custom scripts during upgrade 29 Hortonworks Inc All Rights Reserved

Stack Downgrade Can trigger downgrade at any stage of the stack upgrade Cannot downgrade

33 Hadoop ConfiguraKon Challenges Ã Too many configuraqons Which ones are important? Ã Too easy to mess up What are valid/reasonable values? What are the units? Ok, what about dependencies? Ã Gets harder with combinaqons of services, host assignments, enabled features, CPU/RAM/disks, etc Any recommendaqons? What am I doing wrong? Ã Smart ConfiguraQons 33 Hortonworks Inc All Rights Reserved

34 Ambari Smart Configs UI Customizable layout - Tabs - SecQons - Sub- secqons - Simple grid layout (Advanced Tab contains remaining configuraqons) Implemented - HDFS - YARN - MapReduce - Hive - HBase New Widgets - Sliders - Recommended - Minimum - Maximum - Increment Step - Combos - Enumerated values - Toggles - Binary opqons - Spinners - Splits value into mulqple controls. Time in milliseconds split into days, hours, minutes. - Lists - Enumerated values - Single select - MulQ select 34 Hortonworks Inc All Rights Reserved

35 Stack Driven Layouts Stack has theme.json file Layout Tabs SecQons Sub- secqons Placement Configs placement in sub- secqons Widgets Widget type OpQonal Units Bytes (B, KB, MB, GB, TB, PB) Time (Millis, Seconds, Minutes, Hours, Days, Months, Years) { } "name": "default", "description": "Default theme for HBASE service", "configuration": { "layouts": [ { "name": "default", "tabs": [ { "name": "settings", "display-name": "Settings", "layout": { "tab-columns": "3", "tab-rows": "3", "sections": [... ] } } ] } ], "placement": { "configuration-layout": "default", "configs": [...] }, "widgets": [ { "config": "hbase-env/hbase_master_heapsize", "widget": { "type": "slider", "units": [ { "unit-name": "GB" } ] } },... ] } 35 Hortonworks Inc All Rights Reserved

36 Config Metadata and Dependencies Extended Metadata Defined in property_value_aaributes Hold non- UI metadata about value range, increment, unit, etc Dependencies Models bi- direcqonal relaqonship between configs Depends On (property_depends_on) Answers which configs do I depend on? Depended By (dependencies) Answers which configs are dependent on me? Ambari automaqcally updates dependencies { } "StackConfigurations": { "final": "false", "property_depends_on": [ { "type": "yarn-site", "name": "yarn.nodemanager.resource.memory-mb" } ], "property_description": The minimum allocation for every", "property_display_name": "Minimum Container Size (Memory)", "property_name": "yarn.scheduler.minimum-allocation-mb", "property_type": [], "property_value": "512", "property_value_attributes": { "type": "int", "maximum": "5120", "minimum": "0", "unit": "MB", "increment_step": "256" }, "type": "yarn-site.xml" }, "dependencies": [ { "StackConfigurationDependency": { "dependency_name": "hive.tez.container.size", "property_name": "yarn.scheduler.minimum-allocation-mb } }, { "StackConfigurationDependency": { "dependency_name": "mapreduce.map.memory.mb", "property_name": "yarn.scheduler.minimum-allocation-mb } }, { "StackConfigurationDependency": { "dependency_name": "mapreduce.reduce.memory.mb", "property_name": "yarn.scheduler.minimum-allocation-mb } } ] 36 Hortonworks Inc All Rights Reserved

38 Ambari Metrics Service (AMS) - Goals Ã Ability to collect metrics from Hadoop and other Stack services Ã Ability to collect system level metrics Ã Ability to retain metrics at a high precision for a configurable Qme period Ã Ability to automaqcally purge metrics aoer retenqon period Ã Provide integraqon point for metrics collecqon and retenqon by external system Ã Trigger alerts based on metrics in Ambari 38 Hortonworks Inc All Rights Reserved

40 AMS Grafana Ambari Ã Powerful dashboard builder integrated with AMS Ã Pre- built Grafana dashboards for host- level and service- level metrics Ã User can build and save custom dashboards 40 Hortonworks Inc All Rights Reserved

43 Alert Types Type DescripKon Status PORT Watches a port based on a configuraqon property such as the URI. OK, WARN, CRIT Thresholds Configurable? Yes (seconds) WEB Watches an HTTP or HTTPS endpoint and determines connecqvity and HTTP status code. OK, WARN, CRIT No AGGREGATE Aggregate of status for another alert definiqon. OK, WARN, CRIT Yes (percentage) METRIC Watches a metric or series of metrics in JMX and compares a mathemaqcal result against a threshold. OK, WARN, CRIT Yes (variable) SCRIPT Uses a custom script to handle checking. OK or CRIT No 43 Hortonworks Inc All Rights Reserved

UI Current Alerts Configured by default; managed via the the

UI Host Alerts AutomaQcally refreshes Query alert

48 Ambari Views View Framework Provide various applicaqons accessible from Ambari Web UI interact with the cluster via a browser from a single place for all users (cluster operators, data analysis, developers, etc) Easy to develop No need to understand Ambari core code view development is just like creaqng any other web applicaqon Easy to deploy Packaged as a single jar file Auto create / auto configure 48 Hortonworks Inc All Rights Reserved

Query Editors for Data Analysts Create, edit, execute, and analyze Hive queries

53 Ambari Server in Views- Only mode Ã Ã Use Views on exisqng clusters not managed by Ambari Can use Views against mulqple clusters Ambari Server Management Use Views Cluster managed by Ambari Use Views Cluster not managed by Ambari Use Views Ambari Server Views- only mode (aka Stand- alone mode) 53 Hortonworks Inc All Rights Reserved

55 Kerberos AutomaKon Ambari 2.0 Ã Ambari manage Kerberos principals and keytabs Ã Works with exisqng MIT KDC or AcQve Directory Ã Once Kerberized, seamlessly handle: Ã Adding new hosts Ã Adding new components to exisqng hosts Ã Adding new services Ã Moving components to different hosts 55 Hortonworks Inc All Rights Reserved

58 Thank You! Try Ambari Ã Follow the Ambari Quick Start Guide hkps://cwiki.apache.org/confluence/display/ambari/quick+start+guide Learn more Ã Visit the project website hkp://ambari.apache.org/ Get Involved Ã Ã Ã User Mailing List: user- Developer Mailing List: dev- Use JIRA to file bugs and improvement requests hkps://issues.apache.org/jira/browse/ambari/ Jayush Hortonworks (Apache Ambari PMC) 58 Hortonworks Inc All Rights Reserved

60 Q&A Stats Largest producqon clusters managed by Ambari Largest test cluster for Ambari scale tesqng Largest test cluster where rolling upgrade was performed ~1600 nodes, ~800 nodes ~400 nodes ~400 nodes ~40 hours 60 Hortonworks Inc All Rights Reserved

Setup HSP Ambari Cluster

Setup HSP Ambari Cluster Prerequisites 1. An initialized HSP cluster running at least HSP 1.1.1. 2. Downloaded copy of the ISO containing the vm-template for Ambari from HortonWorks. This ISO is available