Rapid Start with Big Data Appliance X6-2 Technical & Operational Overview

Size: px

Start display at page:

Download "Rapid Start with Big Data Appliance X6-2 Technical & Operational Overview"

Elijah Ball
5 years ago
Views:

1 Rapid Start with Big Data Appliance X6-2 Technical & Operational Overview Dirk Augustin Solution Architect Hardware Presales Germany

2 The Realities of Today s Data Center... Accelerating Customer Expectations Instantly Available Services Accelerating Information Growth Explosion of Data Volume and Types Accelerating Business Complexity Unpredictability and Rapid Change Wherever They Are Transaction Overload Ecosystem Interdependency 2

Business: How can I realize my use case? 2012 Is the design and build of a Big Data Platform our competence and business goal? Do we have the Time to Build?

3 Business: How can I realize my use case? 2012 Is the design and build of a Big Data Platform our competence and business goal? Do we have the Time to Build? Do we want to invest into high development Costs and Difficulty Maintaining? What are the operational aspects and what are the costs behind them? Lower TCO and Faster Time to Value with Oracle Engineered Systems! 3

4 Agenda - Oracle Big Data Appliance Platform overview Operational Simplicity Installation, Upgrade / Patching, ASR, Platform Management High Availability, Data Consistency and Security Oracle s Engineered Big Data Solution Platform 4

5 Oracle Big Data Appliance (BDA) - X6-2 An engineered system of hardware and software optimized to capture and analyze the massive volumes of unstructured data A high performance, secure platform for running diverse workloads on Hadoop and NoSQL systems Engineered to work with Oracle Exadata Database Machine and Oracle Exalytics In-Memory Machine to provide the most advanced analysis of all data types, with enterprise-class performance, availability, supportability, and security. Copyright 2016 Oracle and/or its affiliates. All rights reserved. Data Vision

2GHz) Intel Xeon E5-2699 v4 Processors 256 GB DDR4-2400 (max 768 GB) Memory 96TB Disk local

6 Infiniband Network Oracle Big Data Appliance X6-2 - Hardware Sun Oracle X6-2L Servers with per server: 2 * 22 Core (2.2GHz) Intel Xeon E v4 Processors 256 GB DDR (max 768 GB) Memory 96TB Disk local Storage Capacity Latest Hardware Technologies Up to 13,8 TB available in FullRack 6

7 Oracle Big Data Appliance X6-2 - Software Included Software ( MBSW Version 4.7 / CDH 5.9.0) Oracle Linux 6.7 Cloudera Distribution of Apache Hadoop (Enterprise Data Hub Edition) Cloudera Manager Oracle R Distribution Oracle NoSQL Database CE Oracle Big Data SQL (optional, separately licensed) Cloudera officially certifies BDA as an officially certified partner product (as well as Big Data Cloud Service) The complete integration work is done for you so new BD services come online quickly Oracle Big Data Appliance eliminates data silos, quickens data processing, and simplifies management Copyright 2016 Oracle and/or its affiliates. All rights reserved. Data Vision

8 Elastically Scale-Out from Starter Rack to Multi-Rack HC Starter Full Start with 6 BDA Servers and all switches - Add BDA HC Nodes as needed Can expand older machines with new generation servers Multi-Rack 8

9 Operational Simplicity Network integration Oracle Big Data Appliance (Doc ID ) 10

10 Operational Simplicity Network integration Oracle Big Data Appliance (Doc ID ) 11

Operational Simplicity Configuration & Patching Easy and fast Full Stack installation / configuration BDA Configuration Utillity The Oracle Big Data Appliance Configuration Utility is used to

11 Operational Simplicity Configuration & Patching Easy and fast Full Stack installation / configuration BDA Configuration Utillity The Oracle Big Data Appliance Configuration Utility is used to generate the installation and deployment files These files help automate the deployment process and ensure that Oracle BDA is configured to your specifications. HW Config SW Config BDACLI Utility (Command Line Tool) information about the rack, cluster, server, InfiniBand network, and software patches Easy and fast SW installation / patching Mammoth is a command-line utility for installing and configuring the Oracle Big Data Appliance Software Create one or multiple CDH cluster or Oracle NoSQL Database cluster on one or more BDA racks Extend a CDH cluster on to new servers Update, Patch a cluster (Cloudera stack with all components ) Deploy ASR (Auto Service Request) during or after the initial software installation Deploy the Oracle Enterprise Manager system monitoring plugin for Oracle Big Data Appliance during or after the initial software installation Master note: DocID BDA Mammoth bundle releases follow the Cloudera release schedule. These bundles are tested by Oracle before released OBDA Owners Guide: Full install of BDA Software with a single command:./mammoth i bdarack_1 12

Operational Simplicity Configuration & Patching Easy and fast HW Upgrade with no downtime Hadoop Cluster expansion with a single command: mammoth e newhost1,,newhostn This expansion automatically

12 Operational Simplicity Configuration & Patching Easy and fast HW Upgrade with no downtime Hadoop Cluster expansion with a single command: mammoth e newhost1,,newhostn This expansion automatically optimizes HA setup across multiple racks, e.g. Hadoop Name Node Because of uniform nodes and IB networking, no data is moved Easy and fast automatic Service - ASR (Option) ASR monitors the health of Oracle Big Data Appliance hardware and automatically submits a service request when it detects a fault Copyright 2016 Oracle and/or its affiliates. All rights reserved. Data Vision

Operational Simplicity Monitoring / Management OEM Oracle HW Management The BDA EM plugin allows organizations to manage the BDA using a consistent OEM management framework Use the plug-in to monitor

13 Operational Simplicity Monitoring / Management OEM Oracle HW Management The BDA EM plugin allows organizations to manage the BDA using a consistent OEM management framework Use the plug-in to monitor both hardware and software performance, manage incidents and install/configure BDA options Integrates with and complements Cloudera Manager - which provides detail Hadoop cluster configuration OEM Oracle SW Management OS NoSQL Other Oracle products OEM CDH Management Cluster Growth Node Migration Monitoring BDA Exachk Doc ID ( and Doc ID ) can audit important configuration settings within a BDA. Exachk examines the following components: Compute CPU Hardware, Firmware, BIOS Operating System - kernel parameters, system packages Network - Ethernet, InfiniBand Memory - RAM, disks Software Installed 14

14 Operational Simplicity Monitoring / Management The Oracle Enterprise Manager Big Data Appliance (BDA) Plug-in enables you e.g. to: monitor and manage the BDA Rack Hardware monitor and manage the BDA Software (e.g. Hadoop) monitor and manage the BDA Mammoth Utility to improve quality of service reimage BDA clusters and add a new NoSQL Oracle clusters execute configuration compliance checks monitor a collection of new map-reduced and HDFS metrics such as Jobs / Tasks Failed, Jobs Killed, JobTracker Alert Rate, TaskTracker Alert Rate, BUT: Customer Admin need Unix/Linux- and CDH Know How > Oracle ACS or Oracle Partner could help! Oracle ACS - Advanced Support for Oracle Big Data Appliance 15

15 Cloudera Software BDA Hardware Monitoring / Management BDA Hardware 16

16 Here is the software overview across clusters. You can see what services are running across the nodes and their status Copyright 2016 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 17

added data consistency Oracle Big Data Appliance

17 Big Data Appliance HA with data consistency Big Data Appliance replication ensures high availability and added data consistency Oracle Big Data Appliance Maximum Availability Architecture WhitePaper 03/

18 Big Data Appliance Always High Available and Secure Hardware Software Server / Storage Network Rack PDUs NameNodes Resource Managers NoSQL Database Cloudera Manager Oozie server Hive server Hue server Oracle Data Integrator agent HA with automatic failover / Redundant Component Redundant Redundant Redundant X X HA with NO automatic failover X Restart manual on another Server X X X X X 19

19 Big Data Appliance Always Highly Available and Secure CDH cluster MIT/AD Kerberos Authentification and Auditing MIT and Active Directory Kerberos authentication are security options for OEM and CDH clusters Oracle Audit Vault (Optional) provides an integrated auditing platform for your enterprise. It makes the auditing information for BDA available in a single reporting framework. HDFS: Who makes changes to the file system. Hive DDL: Who makes Hive database changes. MapReduce: Who runs MapReduce jobs that correspond to file access. Oozie workflows: Who runs workflow activities. E.g. EU-Datenschutz-Grundverordnung (DSGVO) Informationspflicht -> transparente DV HDFS Transparent- and HTTPS/Network Encryption HDFS Transparent Encryption protects Hadoop data that is at rest on disk. Data writes and reads are automatically encrypted and decrypted. This process is transparent for applications working with the data HDFS Transparent Encryption does not affect user access to Hadoop data, although it can have a minor impact on performance Oracle recommendation: Navigator Key Trustee (the service that manages keys and certificates) should be on a separate server, external to the Oracle Big Data Appliance Web Interface Encryption: Configures HTTPS for Cloudera Manager, Oozie, and HUE Encrypt Hadoop Services and HDFS Data Transport 20

20 Oracle s Big Data Solution since many years Oracle Big Data Appliance Oracle Exadata Oracle Exalytics Infiniband Infiniband Stream Acquire Organize & Discover Analyze Decide 21

21 BDA vs. Build Your Own - Cost comparison example White Paper written by Nik Rouda, Senior Analyst and Adam DeMattia, Research Analyst - December data/eng-systems-for-big-data-esg-wp pdf Copyright 2016 Oracle and/or its affiliates. All rights reserved. Data Vision

Oracle Big Data Appliance Out of the Box Optimized for acquiring, organizing, loading unstructured data into Oracle DB Simplify IT with Big Data Appliance www.oracle.

22 Oracle Big Data Appliance Out of the Box Optimized for acquiring, organizing, loading unstructured data into Oracle DB Simplify IT with Big Data Appliance Benefits Faster Time to Value with a complete and optimized solution for big data Most comprehensive big data tool set integrated in a single appliance easy-to-deploy Leverage SQL-based tools and data warehouse platforms for complete data analysis Single-vendor support for both hardware and software Unique Features High-speed InfiniBand interconnect for moving information to Exadata Super-Fast Massively Parallel Processing & Loading into Oracle DB Copyright 2016 Oracle and/or its affiliates. All rights reserved. Data Vision

23 More links Oracle Big Data Appliance Official Website Oracle Big Data Appliance Documentation Oracle Big Data Lite Virtual Machine 24

HW Verlosung Mittwoch - 15 Uhr Apple ipad Air 2 Copyright 2016

Oracle Big Data Cloud Service

Oracle Big Data Cloud Service Delivering Hadoop, Spark and Data Science with Oracle Security and Cloud Simplicity Oracle Big Data Cloud Service is an automated service that provides a highpowered environment