Bots, Outliers and Outages

Transcription

1 Bots, Outliers and Outages Do you know what's lurking in your data? Matteo Rebeschini Solutions Elastic matteo@elastic.co 2018 Phoenix Data Conference

2 Abstract With the mass amounts of data that are being ingested daily it is nearly impossible by traditional means to understand what is hidden in your data. How do you separate the ordinary from the un-ordinary in a timely fashion? Unsupervised machine learning on time series data enables real-time discovery of those interesting and possibly costly data anomalies. Matteo will describe, build and run several types of machine learning jobs in Elasticsearch that can detect and alert on these anomalies and outliers in real time.

3

4 What s Elastic? 3

5 search 4

6 search 5

7 100, M Meetup Members Product Downloads Subscription Customers Statistics since 2012, founding of Elastic 6

8 Users across all segments

9 The Elastic Stack (AKA The ELK Stack) 8

10

11 How Elastic Stack Components Work Together Beats Logs Metrics Packets... Logstash Datastore Web APIs Elasticsearch Kibana

12 Deployment in the Enterprise Beats Elasticsearch Custom UI FILEBEAT HEARTBEAT WINGLOGBEAT Master (3) Ingest (X) Logstash METRICBEAT Coordinating (X) PACKETBEAT Elasticsearch Clients AUDITBEAT Data Hot (X) Kafka Kibana Data Warm (X) Data store Web APIs Redis Workers (2+) Alerting (X) Messaging Queue Social Machine Learning (2+) Sensors LDAP ES-Hadoop 11 AD SSO Authentication All product names, logos, and brands are property of their respective owners and are used only for identification purposes. This is not an endorsement. SAML Notification

13 Application Search Site Search Enterprise Search Logging APM Business Analytics Security Analytics Future Metrics Solutions Visualize & Manage Store, Search, & Analyze Elastic Stack Ingest SaaS Elastic Cloud Self Managed Elastic Cloud Enterprise Standalone Deployment

14 Machine Learning with the Elastic Stack 13

15 What s Machine Learning? Algorithms that Learn from Data Using Statistical Techniques Without Explicit Programming

16 Elastic Machine Learning Scope Image Classification Recommendations Autonomous cars Voice Recognition Predictive Medicine Fraud detection Anomaly Detection Speech Recognition Language Translation Entity Resolution Learn to Rank

17 Elastic Machine Learning Scope Method Elastic ML Supervised Unsupervised Anomaly Detection Panel Time Series Classification Problem Cross Section Regression Data

18 Elastic Machine Learning Flow

19 Challenges that Anomaly Detection Solves IT Operations How do I know my systems are behaving normally? Where to set thresholds for good alerting? How to find the root cause of problems when I don t know what to look for? IT Security Do I have systems that are compromised with malware? Which users could be an insider threat? IoT / SCADA / Other Is my factory working normally? What do I do with thousands of time-series data points? Which traffic incidents are causing the most delay? 18

20 Detecting (noteworthy) anomalies is hard! Data is complex, high dimensional, fast moving Human inspection is not practical Easy to miss things Where s the anomaly? Visual inspection is not practical 19

21 Detecting (noteworthy) anomalies is hard! Defining normal via static thresholds is hard Rules don t evolve with data / infrastructure Rules can be bypassed What s the right threshold? Rule-based alerts are insufficient 20

22 Elastic Machine Learning Uses unsupervised machine learning techniques to Learn what s normal by modeling historic behavior Detect anomalies when data falls outside expected bounds Use models to predict future behavior (prediction) Use predictions to make decisions 21

23 Elastic Machine Learning Unsupervised techniques - no manual training / input needed Evolves with the data - online model learns continuously Influencer detection - accelerates root cause identification 22

24 Detect anomalies of different types Time series - single / multiple metric(s) Outliers in population (using entity profiling) Rare / unusual events 23

25 Demo 24

26 Technology Behind Elastic ML The technology behind Elastic s Machine Learning (ML) is a bespoke amalgamation of different machine learning methods and techniques that brings sophisticated real-time automated anomaly detection for time series data to users that may not be able to employ data science on their own. Using techniques such as clustering, various types of time-series decomposition, Bayesian distribution modeling, and correlation analysis, Elastic Machine Learning takes a 100% unsupervised machine learning approach to statistically model data s time-based characteristics merely by observing its historical behavior. Behind the scenes, a dynamic, ever-learning statistical model is built and stored, per unique time-series. Real-time data being analyzed both contributes to this model s maturity and is assessed against the model so that it can be judged for its level of unusualness. If the data s behavior is seen as being within the low probability range, an anomaly record is created, persisted, and scored proportional to the probability. This score is normalized on a user-friendly dynamic scale between 0 and 100, where 100 is the most unusual thing ever detected for the data set.