Uncovering the Hidden Truth In Log Data with vcenter Insight

Transcription

1 Uncovering the Hidden Truth In Log Data with vcenter Insight April 2014 VMware vforum Istanbul 2014 Serdar Arıcan 2014 VMware Inc. All rights reserved.

2 VMware Strategy To help customers realize the promise of the Software-Defined Data Center, VMware introduced the Cloud Suite Software-Defined Data Center Virtual Datacenter: The ideal infrastructure for private, public, and hybrid clouds. Hybrid Cloud Computing Extend your data center into the cloud. End User Computing Connectivity & Control: Desktop, App, & Mobile Virtualization & Provisioning 2

3 VMware is Going Bigger on Management! Business momentum 40% bookings growth YoY Accelerating Sphere installed base penetration ~2.5x more customers in last year! Partner commitment 3,000+ selling Mgmt (200% YoY growth) 2,500+ engaged in Mgmt competency Partners invested in Mgmt have grown bookings 200% more than noncompetency partners Market leadership Ranked #1 by IDC, 451 Research and Info-Tech

4 VMware Cloud Management The Control Plane for the Software-Defined Data Center and the Hybrid Cloud Automation Operations IT Business Service Catalog Governance Release Automation Service Health Capacity Optimization Configuration Standards Cost Transparency Benchmarking Service Quality Applications Traditional Modern SaaS Virtualized Infrastructure Abstract & Pool Hybrid Cloud VMware & vcloud Datacenter Partners Compute Abstraction = Server Virtualization Network Abstraction = Virtual Networking Storage Abstraction = Software-Defined Storage Private Clouds Public Clouds Physical Hardware Compute Network Storage 4

5 Why VMware Cloud Management is Better 1 Most complete, innovative management portfolio purpose-built for cloud Scalable solutions for dynamic, hybrid cloud environments Better automation driven by analytics, policy and vcloud / SDDC integration Converged disciplines of performance, capacity and configuration management Fast time to value Easy. Fast deployment and administration Unencumbered by huge and complex legacy management frameworks Best for vsphere, yet open and heterogeneous Co-developed with vsphere team with deep insight into platform Multi-hypervisor/platform management. Works across hundreds of public cloud providers From the trusted market leader Virtualization pioneer and Cloud Systems Management market leader* The only cloud era management vendor with the vision, scale, portfolio breadth and track record to deliver comprehensively to the enterprise * Per IDC based on 2012 Cloud Systems Management market share

6 VMware Cloud Management Leadership 1 st in Cloud Systems Management Software Market Share - IDC Champion in CMP Vendor Landscape - Info-Tech 1 st in four of six Cloud Management and Automation categories: Self-service Catalog Unified Cloud Management Console Cloud Governance Metering and Billing Research

7 Log Insight Overview

8 What Could You Do If You Had Insight? Get actionable insight into what it means vcenter Log Insight Solve problems faster from days to hours Find problems you didn t know you had 8

9 Our Vision Create an intuitive and fast log management platform for cloud operations that delivers proactive analytics through machine learning 9

10 vcenter Log Insight 2.0 Overview Overview Intelligent Operations Predictive analytics/intelligent Grouping for faster problem resolution Faster analytical queries than the leading solution 1 Improved analytical visualizations Built with vsphere in Mind Powerful Log Management for VMware Products Support for Vmware products incl. NSX, vcac, Horizon View Built in 2 way alerting with vc Ops Unified Management Open and extensible platform/marketplace for content packs What s New New! Delivers the best real-time log management for VMware environments, across physical, virtual, and cloud environments. Elastic Scale up to 6 node clusters Improved Query Performance Up to 6X faster than leading solution Improved Data Collection 30% improvement on ingestion Predictive Analytics Machine Learning based Intelligent Grouping Better visualization New Analytic Visualizations Integration with vc Ops 2-way alert visualization Built by VMware Experts vsphere analytics built in Predictable Pricing No surprises on storage costs

11 Log Insight Technical Overview Analyze Can analyze any unstructured time-series data, configuration etc. Automatically identifies structures in the data No need for ETL or databases Scale Central, scale-out store (no-sql) for all collected logs Configurable retention and archiving Maintenance free Log Management API Syslog Best for vsphere Queries, alerts, fields, charts in the vsphere Content Pack OS Logs VC Logs System Stats App Logs Security Logs Cloud / Data Center

12 Log Insight UI - Interactive Log Analytics Interactive Visualization of Query Results, Plus Easy Drop-Down Menu Options

13 Log Insight Features Proactive Analytics Significantly reduces manual parsing effort Automated data summarization Logs are clustered by event type, so rare messages can be easily spotted Intelligent schema detection adds structure to unstructured data Smart fields to aid in extraction Interactive Analytics Google-like search query capability Filter events on any criteria, on any field Flexible event views On-the-fly visualizations 13

14 vcenter Operations and Log Insight Leverage all your IT data for comprehensive visibility in one place VMware vcenter Operations Capacity, Performance and Configuration Management Launch in Context Events VMware vcenter Log Insight Log analytics, aggregation, and search Structured Data Unstructured Data Metrics Alerts Events Logs Messages Public Cloud Intelligent operations through predictive analytics across all machine data Policy-based automation enables proactive management and automated remediation Unified management for comprehensive visibility in one place, from vsphere to Hyper-V, AWS and physical infrastructure

15 Scale-out Architecture With Ingestion High Availability Web UI/Query ( Syslog CFAPI (TCP/UDP:514) ( Load Balancer (UDP & TCP) Log messages (Syslog:514 and Master Node Log Insight Worker Nodes

16 Proactive Analytics

17 Log Insight Proactively Discovers Structure Through Machine Learning! Log Insight proactively learns: from: Then you can query it like a database!

18 Dashboard Enhancements Greatly Improve Effectiveness as Debugging Tools Dashboard Filters Can now apply a filter to all charts on a dashboard page Speeds up investigation of individual entities (hosts) or groups (clusters) Dashboard Linking Hyperlink from a value in one chart to a dashboard constraint in another Log Insight automatically discovers appropriate links Both Features Will Significantly Increase Content Pack Capabilities Richer workflows

19 Windows Agent

20 Collection Framework & API REST-based HTTP API For sending logs to Log Insight For clients to request configuration data Has all the benefits of an HTTP API (compression, proxies, easy to use, etc) Windows Agent VERY SMALL : 3.5MB RAM and 0.4% of 1 CPU 100 events/s Deployable via standard Windows management (.msi) Monitors Windows Events Monitors text file logs (like tail f on Unix/Linux)

21 Where to Go Next Communities Experts: Download Link:

22 Questions?

23 Log Insight is the Right Solution For: Gathering, Analyzing, & Searching large amounts of unstructured data Real-time troubleshooting that requires quick answers VMware environments VMware experts have curated troubleshooting dashboards Native support for vcops & vsphere Taming a large, complex infrastructure

24 Log Insight Features Deploys as a Virtual Appliance Scale-out architecture with high availability One deployment of 6 nodes can handle approx 45K events/second Microsoft Active Directory support Accepts logs over Syslog or RESTful API Any syslog agent can forward logs to Log Insight Lightweight Windows agent for forwarding logs is available 24

25 Primary Use Cases Troubleshooting and Root Cause Analysis Follow the trail from vcops to logs to get to root cause to an observed problem I observed a problem (e.g. slowness), try to troubleshoot the problem and identify the part of the stack that is responsible (e.g. network delay vs storage) Identify the needle in the haystack in real time when troubleshooting a problem Monitoring Monitor metrics and events (performance & change) that are visible only in logs Escalate alerts to on-call staff or via vcops Identify problems proactively, ensure SLAs and comply to IT policies Unstructured Data Warehouse Collect all the data in one place without the need for custom parsing, transformation of data Get full visibility across all your IT environment from a single place 25

26 Log Insight Features Archiving for long-term data storage Super-Powered Dashboards Dashboards called Content Packs are curated by experts for top applications & devices Add custom dashboards & share them with colleagues Beautiful visualizations Easily add new data filters on the fly Interact between dashboard widgets Native support for VMware products Simple to configure ESXi hosts vcenter Operations Manager interactive capabilities 26

27 Log Insight + vcenter Operations Manager Automated correlation of performance and log data (Requires vcenter Operations Advanced or Enterprise) 27

28 Log Insight + vcenter Operations Manager Launch Log Insight from vcenter Operations in-context Escalate alerts & events directly to vcenter Operations Much more on its way! 28