A New Product: Hunk Splunk AnalyBcs for Hadoop (BETA)

Size: px

Start display at page:

Download "A New Product: Hunk Splunk AnalyBcs for Hadoop (BETA)"

Ross Gregory
6 years ago
Views:

2 Legal NoBces During the course of this presentabon, we may make forward- looking statements regarding future events or the expected performance of the company. We caubon you that such statements reflect our current expectabons and esbmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in this presentabon are being made as of the Bme and date of its live presentabon. If reviewed ayer its live presentabon, this presentabon may not contain current or accurate informabon. We do not assume any obligabon to update any forward- looking statements we may make. In addibon, any informabon about our roadmap outlines our general product direcbon and is subject to change at any Bme without nobce. It is for informabonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligabon either to develop the features or funcbonality described or to include any such feature or funcbonality in a future release. Splunk, Splunk>, Splunk Storm, Listen to Your Data, SPL and The Engine for Machine Data are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respeccve owners Splunk Inc. All rights reserved. 2

3 Announcing Hunk Beta Splunk AnalyBcs for Hadoop New product from Splunk delivers interac(ve data explora(on, analysis and visualiza(ons for Hadoop 3

4 A Lot of OrganizaBonal Data Ends Up in Hadoop Challenges Deploying and Leveraging Hadoop! 20X services relabve to soyware! Inadequate skills for big data analybcs! 13+ Hadoop- related projects requiring integrabon Chukwa 13+ Hadoop- related projects HBase Avro Mahout Pig ZooKeeper Cassandr Ambari a H i v e YARN! Data is too big to move Hadoop (MapReduce & HDFS) 4

October 2012: Splunk Hadoop Connect To Address Common Challenges Deploying and Running Hadoop Splunk Hadoop Connect Ad hoc search Monitor and alert Report and analyze

5 October 2012: Splunk Hadoop Connect To Address Common Challenges Deploying and Running Hadoop Splunk Hadoop Connect Ad hoc search Monitor and alert Report and analyze Custom dashboards Reliable bi- direcbonal integrabon to Hadoop >1000 downloads HA indexes and storage Commodity servers Import Browse Export Hadoop (MapReduce & HDFS) 5

and alert Report and analyze Custom dashboards nabvely on data in Hadoop?

6 What About ExtracBng Value Directly from Hadoop? How can we leverage the full capabilibes of Splunk Ad hoc search Monitor and alert Report and analyze Custom dashboards nabvely on data in Hadoop? HA indexes and storage Commodity servers Hadoop (MapReduce & HDFS) Data in Hadoop is too big to move 6

groups to derive acbonable insights from raw data in Hadoop Works with leading distribubons to maximize

7 Hunk: Splunk AnalyBcs for Hadoop Full- featured, integrated product Insights for everyone Distribu(on agnos(c Delivers interacbve data explorabon, analysis and visualizabon for Hadoop Empowers broader user groups to derive acbonable insights from raw data in Hadoop Works with leading distribubons to maximize enterprise technology investments Explore Analyze Visualize Dashboards Share Hadoop (MapReduce & HDFS) 7

8 Derive AcBonable Insights from Raw Data 1 2 Point Splunk at Hadoop cluster Explore Analyze Visualize Dashboards Share Immediately start exploring, analyzing and visualizing raw data in Hadoop Hadoop storage 8

MapReduce Technology is patent pending Structure applied at search Bme No brille schema to work around AutomaBcally find

9 Explore, Analyze and Visualize Data, On- the- fly Virtual index Schema- on- the- fly Flexibility and fast (me to value Enables seamless use of the enbre Splunk technology stack on data wherever it rests Hadoop virtual index automabcally handles MapReduce Technology is patent pending Structure applied at search Bme No brille schema to work around AutomaBcally find transacbons, palerns and trends NormalizaBon as it s needed Faster implementabon Easy search language MulBple views into the same data 9

InteracBve Data ExploraBon Search and explore data from one place Search assistant! Powerful search processing language (SPL) Search interface!

10 InteracBve Data ExploraBon Search and explore data from one place Search assistant! Powerful search processing language (SPL) Search interface! Designed for data explorabon across large datasets preview data, iterate quickly! No requirement to understand data upfront InteracBve results window 10

11 InteracBve Data Analysis Rapidly analyze and interact with data Formaong opbons! InteracBve analybcs interface! Deep analysis, palern detecbon and finding anomalies with over 100 stabsbcal commands! Enrich results with informabon from external relabonal databases InteracBve, analybcs interface 11

12 Powerful Plaporm for Enterprise Developers Add new UI components Integrate into exisbng systems With known languages and frameworks JavaScript Java Python PHP C# Ruby API 12

13 Technology Overview

head Virtual indexes C++, web services COMMAND LINE splunkd 64- bit

14 Hunk Server Explore Analyze Visualize Dashboards Share splunkweb Web and applicabon server Python, AJAX, CSS, XSLT, XML REST API Search head Virtual indexes C++, web services COMMAND LINE splunkd 64- bit Linux OS 14 ODBC (beta) Hadoop interface Hadoop client libraries JAVA

15 Connect to HDFS and MapReduce Explore Analyze Visualize Dashboards Share splunkweb Web and applicabon server Python, AJAX, CSS, XSLT, XML REST API Search head Virtual indexes C++, web services COMMAND LINE splunkd ODBC (beta) Hadoop interface Hadoop client libraries JAVA Connect to Apache HDFS and MapReduce or your choice of Hadoop distribubon Hadoop cluster bit Linux OS 15

16 Hunk Scales with Your Hadoop Deployments Explore Analyze Visualize Dashboards Share splunkweb Web and applicabon server Python, AJAX, CSS, XSLT, XML REST API Search head Virtual indexes C++, web services COMMAND LINE splunkd ODBC (beta) Hadoop interface Hadoop client libraries JAVA Connect Hunk to mulbple Hadoop clusters Hadoop cluster 1 Hadoop cluster 2 Hadoop cluster bit Linux OS 16

17 Prerequisites Data in Hadoop to analyze Hadoop client libraries Hadoop access rights Java 1.6+ HDFS scratch space DataNode local temp disk space 17

tgz TaskTracker 1 TaskTracker 2 TaskTracker 3 3.

18 MapReduce as The OrchestraBon Framework Hunk search head > 1. Copy splunkd binary.tgz.tgz HDFS 2. Copy.tgz TaskTracker 1 TaskTracker 2 TaskTracker 3 3. Expand in specified locabon on each TaskTracker 4. Receive binary in subsequent searches 18

19 Hunk Usage in HDFS hdfs://<scratch_space_path>/ bundles Search head bundles: keeps last 5 bundles packages Hunk.tgz packages: no automabc cleanup dispatch/<sid> Search scratch space: cleanup when sid is invalid 19

20 Hunk Uses Virtual Indexes! Enables seamless use of almost the enbre Splunk stack on data in Hadoop! AutomaBcally handles MapReduce! Technology is patent pending 20

21 Examples of Virtual Indexes External system 1 index = syslog (/home/syslog/ ) Hunk search head > External system 2 index = apache_logs index = sensor_data External system 3 index = twiler 21

22 Define Virtual Indexes and Paths External resource (e.g. hadoop.prod) Virtual index (e.g. twiler) Virtual index (e.g. sensor data) Virtual index (e.g. Apache logs) Specify virtual index and data paths, and opbonally:! Filter files or directories using a whitelist or blacklist! Extract metadata or Bme range from paths! Use props/transforms.conf to specify search Bme processing 22

23 Search Data in Hadoop Run a copy of splunkd to process Hunk search head > NameNode DataNode / TaskTracker (Node in YARN) / working directory 1 JSON configs External resource (e.g. hadoop.prod) 5 2 MapReduce jobs JobTracker (MapReduce resource manager in YARN) Tasks 3 DataNode / TaskTracker (Node in YARN) DataNode / TaskTracker (Node in YARN) 4 HDFS 23

e.g. Apache Avro or format readers Event breaking Timestamping Event

24 Data Processing Pipeline Raw data (HDFS) Custom processing stdin Indexing pipeline Search pipeline You can plug in data preprocessors e.g. Apache Avro or format readers Event breaking Timestamping Event typing Lookups Tagging Search processors MapReduce/Java splunkd/c++ 24

25 Hunk Applies Schema on The Fly Structure applied at search Bme No brille schema to work around AutomaBcally find palerns and trends Hunk applies schema for all fields including transacbons at search Bme 25

26 Search OpBmizaBon: ParBBon Pruning! Most data types are stored in hierarchical directories Such as /<base_path>/<date>/<hour>/<hostname>/somefile.log! You can instruct Hunk to extract fields and Bme ranges from a path! Searches ignore directories that cannot possibly contain search results Such as Bme ranges outside of a defined range Example Bme- based parbbon pruning Search: index=hunk earliest_(me= T01:00:00 latest_(me = T02:00:00 26

27 Search Performance with MapReduce MapReduce considerabons! Stats/chart/Bmechart/top/etc. commands work well in a distributed environment They MapReduce well! Time and order commands don t work well in a distributed environment They don t MapReduce well Summary indexing Useful for speeding up searches Summaries could have different retenbon policy In most cases resides on the search head Backfill is a manual (scripted) process 27

28 Mixed- mode Search Streaming Transfers first several blocks from HDFS to the Hunk search head for immediate processing ReporBng Pushes computabon to the DataNodes and TaskTrackers for the complete search! Hunk starts the streaming and reporbng modes concurrently! Streaming results show unbl the reporbng results come in! Allows users to search interacbvely by pausing and refining queries 28

29 Flexible, IteraBve Workflow for Business Users Interac(ve Analy(cs Preview results NormalizaBon as it s needed Faster implementabon and flexibility Easy search language + data models & pivot MulBple views into the same data Share Visualize Explore Pivot Analyze Model 29

30 Demo

31 Next Steps Download the.conf2013 Mobile App If not iphone, ipad or Android, use the Web App Take the survey & WIN A PASS FOR.CONF2014 Or one of these bags! Go to Technical Deep Dive: Hadoop Opera(ons Management Brera 6, Level 3 Today, 11:30-12:30pm 31

32 Thank You

Faster Bme to Value with ITSI Modules

Copyright 2016 Splunk Inc. Faster Bme to Value with ITSI Modules Nicholas Tankersley Bill Babilon Product Manager, Splunk Sales Engineer, Splunk Disclaimer During the course of this presentabon, we may