Pro Apache Hadoop. Second Edition. Sameer Wadkar Madhu Siddalingaiah

Transcription

1 Pro Apache Hadoop Second Edition Sameer Wadkar Madhu Siddalingaiah

2 Pro Apache Hadoop Copyright 2014 by Sameer Wadkar and Madhu Siddalingaiah This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law. ISBN-13 (pbk): ISBN-13 (electronic): Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Publisher: Heinz Weinheimer Lead Editor: Jonathan Gennick Technical Reviewer: Vimlesh Om Mittal Editorial Board: Steve Anglin, Mark Beckner, Ewan Buckingham, Gary Cornell, Louise Corrigan, Jim DeWolf, Jonathan Gennick, Jonathan Hassell, Robert Hutchinson, Michelle Lowman, James Markham, Matthew Moodie, Jeff Olson, Jeffrey Pepper, Douglas Pundick, Ben Renow-Clarke, Dominic Shakeshaft, Gwenan Spearing, Matt Wade, Steve Weiss Coordinating Editor: Jill Balzano Copy Editor: Nancy Sixsmith Compositor: SPi Global Indexer: SPi Global Artist: SPi Global Cover Designer: Anna Ishchenko Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY Phone SPRINGER, fax (201) , orders-ny@springer-sbm.com, or visit Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please rights@apress.com, or visit Apress and friends of ED books may be purchased in bulk for academic, corporate, or promotional use. ebook versions and licenses are also available for most titles. For more information, reference our Special Bulk Sales ebook Licensing web page at Any source code or other supplementary material referenced by the author in this text is available to readers at For detailed information about how to locate your book s source code, go to

3 This book is dedicated to the Open Source Community whose contributions have made Software Development such as vibrant and exciting profession. Sameer To Sasha, Eashan, and Shilpa. Madhu

4

5 Contents at a Glance About the Authors... xix About the Technical Reviewer... xxi Acknowledgments... xxiii Introduction... xxv Chapter 1: Motivation for Big Data...1 Chapter 2: Hadoop Concepts...11 Chapter 3: Getting Started with the Hadoop Framework...31 Chapter 4: Hadoop Administration...47 Chapter 5: Basics of MapReduce Development...73 Chapter 6: Advanced MapReduce Development Chapter 7: Hadoop Input/Output Chapter 8: Testing Hadoop Programs Chapter 9: Monitoring Hadoop Chapter 10: Data Warehousing Using Hadoop Chapter 11: Data Processing Using Pig Chapter 12: HCatalog and Hadoop in the Enterprise Chapter 13: Log Analysis Using Hadoop Chapter 14: Building Real-Time Systems Using HBase Chapter 15: Data Science with Hadoop Chapter 16: Hadoop in the Cloud v

6 Contents at a Glance Chapter 17: Building a YARN Application Appendix A: Installing Hadoop Appendix B: Using Maven with Eclipse Appendix C: Apache Ambari Index vi

7 Contents About the Authors... xix About the Technical Reviewer... xxi Acknowledgments... xxiii Introduction... xxv Chapter 1: Motivation for Big Data...1 What Is Big Data?...1 Key Idea Behind Big Data Techniques...2 Data Is Distributed Across Several Nodes... 2 Applications Are Moved to the Data... 3 Data Is Processed Local to a Node... 3 Sequential Reads Preferred Over Random Reads... 3 An Example... 4 Big Data Programming Models...4 Massively Parallel Processing (MPP) Database Systems... 4 In-Memory Database Systems... 5 MapReduce Systems... 5 Bulk Synchronous Parallel (BSP) Systems... 6 Big Data and Transactional Systems...7 How Much Can We Scale?...8 A Compute-Intensive Example... 8 Amdhal s Law... 9 Business Use-Cases for Big Data...9 Summary...10 vii

8 Contents Chapter 2: Hadoop Concepts...11 Introducing Hadoop...11 Introducing the MapReduce Model...12 Components of Hadoop...16 Hadoop Distributed File System (HDFS) Secondary NameNode TaskTracker JobTracker Hadoop Components of YARN HDFS High Availability...29 Summary...30 Chapter 3: Getting Started with the Hadoop Framework...31 Types of Installation...31 Stand-Alone Mode Pseudo-Distributed Cluster Multinode Node Cluster Installation Preinstalled Using Amazon Elastic MapReduce Setting up a Development Environment with a Cloudera Virtual Machine...33 Components of a MapReduce program...34 Your First Hadoop Program...34 Prerequisites to Run Programs in Local Mode WordCount Using the Old API Building the Application Running WordCount in Cluster Mode WordCount Using the New API Building the Application Running WordCount in Cluster Mode Third-Party Libraries in Hadoop Jobs...41 Summary...46 viii

9 Contents Chapter 4: Hadoop Administration...47 Hadoop Configuration Files...47 Configuring Hadoop Daemons...48 Precedence of Hadoop Configuration Files...49 Diving into Hadoop Configuration Files...49 core-site.xml hdfs-*.xml mapred-site.xml yarn-site.xml Memory Allocations in YARN Scheduler...56 Capacity Scheduler Fair Scheduler Fair Scheduler Configuration yarn-site.xml Configurations Allocation File Format and Configurations Determine Dominant Resource Share in drf Policy Slaves File...64 Rack Awareness...64 Providing Hadoop with Network Topology Cluster Administration Utilities...65 Check the HDFS Command-Line HDFS Administration Rebalancing HDFS Data Copying Large Amounts of Data from the HDFS Summary...72 Chapter 5: Basics of MapReduce Development...73 Hadoop and Data Processing...73 Reviewing the Airline Dataset...73 Preparing the Development Environment Preparing the Hadoop System ix

10 Contents x MapReduce Programming Patterns...76 Map-Only Jobs (SELECT and WHERE Queries) Problem Definition: SELECT Clause Problem Definition: WHERE Clause Map and Reduce Jobs (Aggregation Queries) Problem Definition: GROUP BY and SUM Clauses Improving Aggregation Performance Using the Combiner Problem Definition: Optimized Aggregators Role of the Partitioner Problem Definition: Split Airline Data by Month Bringing it All Together Summary Chapter 6: Advanced MapReduce Development MapReduce Programming Patterns Introduction to Hadoop I/O Problem Definition: Sorting Problem Definition: Analyzing Consecutive Records Problem Definition: Join Using MapReduce Problem Definition: Join Using Map-Only jobs Writing to Multiple Output Files in a Single MR Job Collecting Statistics Using Counters Summary Chapter 7: Hadoop Input/Output Compression Schemes What Can Be Compressed? Compression Schemes Enabling Compression Inside the Hadoop I/O processes InputFormat OutputFormat Custom OutputFormat: Conversion from Text to XML

11 Contents Custom InputFormat: Consuming a Custom XML file Hadoop Files SequenceFile MapFiles Avro Files Summary Chapter 8: Testing Hadoop Programs Revisiting the Word Counter Introducing MRUnit Installing MRUnit MRUnit Core Classes Writing an MRUnit Test Case Testing Counters Features of MRUnit Limitations of MRUnit Testing with LocalJobRunner Limitations of LocalJobRunner Testing with MiniMRCluster Setting up the Development Environment Example for MiniMRCluster Limitations of MiniMRCluster Testing MR Jobs with Access Network Resources Summary Chapter 9: Monitoring Hadoop Writing Log Messages in Hadoop MapReduce Jobs Viewing Log Messages in Hadoop MapReduce Jobs User Log Management in Hadoop 2.x Log Storage in Hadoop 2.x Log Management Improvements Viewing Logs Using Web Based UI xi

12 Contents Command-Line Interface Log Retention Hadoop Cluster Performance Monitoring Using YARN REST APIs Managing the Hadoop Cluster Using Vendor Tools Ambari Architecture Summary Chapter 10: Data Warehousing Using Hadoop Apache Hive Installing Hive Hive Architecture Metastore Compiler Basics Hive Concepts HiveQL Compiler Details Data Definition Language Data Manipulation Language External Interfaces Hive Scripts Performance MapReduce Integration Creating Partitions User-Defined Functions Impala Impala Architecture Impala Features Impala Limitations Shark Shark/Spark Architecture Summary xii

13 Contents Chapter 11: Data Processing Using Pig An Introduction to Pig Running Pig Executing in the Grunt Shell Executing a Pig Script Embedded Java Program Pig Latin Comments in a Pig Script Execution of Pig Statements Pig Commands User-Defined Functions Eval Functions Invoked in the Mapper Eval Functions Invoked in the Reducer Writing and Using a Custom FilterFunc Comparison of PIG versus Hive Crunch API How Crunch Differs from Pig Sample Crunch Pipeline Summary Chapter 12: HCatalog and Hadoop in the Enterprise HCatalog and Enterprise Data Warehouse Users HCatalog: A Brief Technical Background HCatalog Command-Line Interface WebHCat HCatalog Interface for MapReduce HCatalog Interface for Pig HCatalog Notification Interface Security and Authorization in HCatalog Bringing It All Together Summary xiii

14 Contents Chapter 13: Log Analysis Using Hadoop Log File Analysis Applications Web Analytics Security Compliance and Forensics Monitoring and Alerts Internet of Things Analysis Steps Load Refine Visualize Apache Flume Core Concepts Netflix Suro Cloud Solutions Summary Chapter 14: Building Real-Time Systems Using HBase What Is HBase? Typical HBase Use-Case Scenarios HBase Data Model HBase Logical or Client-Side View Differences Between HBase and RDBMSs HBase Tables HBase Cells HBase Column Family HBase Commands and APIs Getting a Command List: help Command Creating a Table: create Command Adding Rows to a Table: put Command Retrieving Rows from the Table: get Command Reading Multiple Rows: scan Command xiv

15 Contents Counting the Rows in the Table: count Command Deleting Rows: delete Command Truncating a Table: truncate Command Dropping a Table: drop Command Altering a Table: alter Command HBase Architecture HBase Components Compaction and Splits in HBase Compaction HBase Configuration: An Overview hbase-default.xml and hbase-site.xml HBase Application Design Tall vs. Wide vs. Narrow Table Design Row Key Design HBase Operations Using Java API HBase Treats Everything as Bytes Create an HBase Table Administrative Functions Using HBaseAdmin Accessing Data Using the Java API HBase MapReduce Integration A MapReduce Job to Read an HBase Table HBase and MapReduce Clusters Scenario I: Frequent MapReduce Jobs Against HBase Tables Scenario II: HBase and MapReduce have Independent SLAs Summary Chapter 15: Data Science with Hadoop Hadoop Data Science Methods Apache Hama Bulk Synchronous Parallel Model Hama Hello World! xv

16 Contents Monte Carlo Methods K-Means Clustering Apache Spark Resilient Distributed Datasets (RDDs) Monte Carlo with Spark KMeans with Spark RHadoop Summary Chapter 16: Hadoop in the Cloud Economics Self-Hosted Cluster Cloud-Hosted Cluster Elasticity On Demand Bid Pricing Hybrid Cloud Logistics Ingress/Egress Data Retention Security Cloud Usage Models Cloud Providers Amazon Web Services Google Cloud Platform Microsoft Azure Choosing a Cloud Vendor Case Study: Amazon Web Services Elastic MapReduce Elastic Compute Cloud Summary xvi

17 Contents Chapter 17: Building a YARN Application YARN: A General-Purpose Distributed System YARN: A Quick Review Creating a YARN Application POM Configuration DownloadService.java Class Client.java Steps to Launch the Application Master from the Client ApplicationMaster.java Communication Protocol between Application Master and Resource Manager: Application Master Protocol Node Manager Communication Protocol: Container Management Protocol Steps to Launch the Worker Tasks Executing the Application Master Launch the Application in Un-Managed Mode Launch the Application in Managed Mode Summary Appendix A: Installing Hadoop Installing Hadoop on Windows Preparing the Installation Environment Building Hadoop for Windows Installing Hadoop for Windows Configuring Hadoop Preparing the Hadoop Cluster Starting HDFS Starting MapReduce (YARN) Verifying that the Cluster Is Running Testing the Cluster Installing Hadoop on Linux xvii

18 Contents Appendix B: Using Maven with Eclipse A Quick Introduction to Maven Creating a Maven Project Using Maven with Eclipse Installing the m2e Maven Eclipse Plug-in Creating a Maven Project from Eclipse Building a Maven Project from Eclipse Appendix C: Apache Ambari Hadoop Components Supported by Apache Ambari Installing Apache Ambari Trying the Ambari Sandbox on Your OS Index xviii

19 About the Authors Sameer Wadkar has more than 16 years of experience in software architecture and development. He has a bachelor s degree in electrical engineering and an MBA in finance from Mumbai University, a postgraduate diploma in software engineering from the National Center of Software Technology (now Center for Development of Advanced Computing), and a master s degree in applied and computational mathematics from Johns Hopkins University. He has implemented distributed systems and high-traffic web sites for a wide variety of clients ranging from federal agencies to investment banking companies. Sameer has been actively working on Hadoop/HBase implementations since 2011 and is also an open-source contributor. His GitHub page is Sameer s open-source contributions include a version of the popular text mining algorithm known as Latent Dirichlett Allocation, which scales to millions of documents on a single machine. He is an avid chess player and plays actively on Madhu Siddalingaiah is a technology consultant with 25 years of experience in a variety of business domains, including aerospace, health care, financial, energy, defense, and scientific research. Over the years, he has specialized in electrical engineering, Internet technologies, and Big Data. More recently, Madhu has delivered several high profile Big Data systems and solutions. He earned his physics degree from the University of Maryland. Outside of his profession, Madhu is a private helicopter pilot, enjoys travel, and participates in a constantly growing list of hobbies. xix

20

21 About the Technical Reviewer Vimlesh Om Mittal has more than 14 years of technology implementation experience and is a Cloudera Certified Developer for Apache Hadoop CDH4. He has a very broad technology background, and his experience includes projects involving modernizing business processes, custom application development, ETL development, digital application development, business intelligence, database design, and data conversions. Vimlesh s experience is a strong blend of architecting robust information management systems, solution architecture, and software development in several development frameworks such as Big Data using the Hadoop ecosystem; data ingestion/acquisition; ETL; reporting/analytics;.net; J2EE; and scripting languages in different vendor technologies such as Cloudera, SAS, IBM, Oracle, Microsoft, ios, and BEA. Vimlesh s focus also includes solutions for financial services data, mobile application design, and development. He is also well-versed in all aspects of the software development life cycle and has extensive experience in project management disciplines. xxi

22

23 Acknowledgments Hadoop has come a long way over the past decade with a large number of vendors and independent developers contributing to it. Hadoop in turn depends on a large number of open source libraries. We would like to acknowledge the effort of all the open source contributors who have contributed to Hadoop. We would also like to acknowledge the blog writers and professionals who contribute to answering Hadoop-related queries on forums. Their participation contributes to making a complex product such as Hadoop mainstream and easier to use. We want to thank the Apress staff members who have applied their expertise to make this book into something readable. We also want to thank our family, friends, and colleagues for supporting us through the writing of this book. xxiii

24

25 Introduction This book is designed to be a concise guide to using the Hadoop software. Despite being around for more than half a decade, Hadoop development is still a very stressful yet very rewarding task. The documentation has come a long way since the early years, and Hadoop is growing rapidly as its adoption is increasing in the Enterprise. Hadoop 2.0 is based on the YARN framework, which is a significant rewrite of the underlying Hadoop platform. It has been our goal to distill the hard lessons learned while implementing Hadoop for clients in this book. As authors, we like to delve deep into the Hadoop source code to understand why Hadoop does what it does and the motivations behind some of its design decisions. We have tried to share this insight with you. We hope that not only will you learn Hadoop in depth but also gain fresh insight into the Java language in the process. This book is about Big Data in general and Hadoop in particular. It is not possible to understand Hadoop without appreciating the overall Big Data landscape. It is written primarily from the point of view of a Hadoop developer and requires an intermediate-level ability to program using Java. It is designed for practicing Hadoop professionals. You will learn several practical tips on how to use the Hadoop software gleaned from our own experience in implementing Hadoop-based systems. This book provides step-by-step instructions and examples that will take you from just beginning to use Hadoop to running complex applications on large clusters of machines. Here s a brief rundown of the book s contents: Chapter 1 introduces you to the motivations behind Big Data software, explaining various Big Data paradigms. Chapter 2 is a high-level introduction to Hadoop 2.0 or YARN. It introduces the key concepts underlying the Hadoop platform. Chapter 3 gets you started with Hadoop. In this chapter, you will write your first MapReduce program. Chapter 4 introduces the key concepts behind the administration of the Hadoop platform. Chapters 5, 6, and 7, which form the core of this book, do a deep dive into the MapReduce framework. You learn all about the internals of the MapReduce framework. We discuss the MapReduce framework in the context of the most ubiquitous of all languages, SQL. We emulate common SQL functions such as SELECT, WHERE, GROUP BY, and JOIN using MapReduce. One of the most popular applications for Hadoop is ETL offloading. These chapters enable you to appreciate how MapReduce can support common data-processing functions. We discuss not just the API but also the more complicated concepts and internal design of the MapReduce framework. Chapter 8 describes the testing frameworks that support unit/integration testing of MapReduce frameworks. Chapter 9 describes logging and monitoring of the Hadoop Framework. Chapter 10 introduces the Hive framework, the data warehouse framework on top of MapReduce. xxv

26 Introduction Chapter 11 introduces the Pig and Crunch frameworks. These frameworks enable users to create data-processing pipelines in Hadoop. Chapter 12 describes the HCatalog framework, which enables Enterprise users to access data stored in the Hadoop file system using commonly known abstractions such as databases and tables. Chapter 13 describes how Hadoop can used for streaming log analysis. Chapter 14 introduces you to HBase, the NoSQL database on top of Hadoop. You learn about use-cases that motivate the use of Hbase. Chapter 15 is a brief introduction to data science. It describes the main limitations of MapReduce that make it inadequate for data science applications. You are introduced to new frameworks such as Spark and Hama that were developed to circumvent MapReduce limitations. Chapter 16 is a brief introduction to using Hadoop in the cloud. It enables you to work on a true production grade Hadoop cluster from the comfort of your living room. Chapter 17 is a whirlwind introduction to the key addition to Hadoop 2.0: the capability to develop your own distributed frameworks such as MapReduce on top of Hadoop. We describe how you can develop a simple distributed download service using Hadoop 2.0. xxvi