Apply Big Data Analytics and Machine Learning in Real Time to Disrupt Business Models. OOP 2017 (Munich, Germany)

Size: px

Start display at page:

Download "Apply Big Data Analytics and Machine Learning in Real Time to Disrupt Business Models. OOP 2017 (Munich, Germany)"

Hortense Eaton
6 years ago
Views:

1 Apply Big Data Analytics and Machine Learning in Real Time to Disrupt Business Models OOP 2017 (Munich, Germany) Kai Wähner Technology Evangelist

2 Apply Big Data Analytics to Real Time Processing

3 Analyze and Act on Critical Business Moments

4 Banking Fraud Detection Anti-Money Laundering (AML) Trade Surveillance Credit/Debit card monitoring Health Insurance Fraud Insurance Fraud Online Operations Steps: 1. Visualise and understand 2. Supervised Learning 3. Unsupervised learning 4. Build and deploy Act proactively before the fraud happened

Partner Network Additional Services Complaint Management

5 Retailer Cross Selling Sale of Premium Upgrade Sale of Perishable Goods Increase of Customer Satisfaction Partner Network Additional Services Complaint Management Act proactively before the customer left the store or location

6 Insurance Context-Specific Pricing Premium Price Quote Optimization Customer Risk Scoring Steps: 1. Build model predicting conversion (churn) with historic quote acceptance and customer attribution data with multiple GLMs Supervised Learning 2. Apply this GLM model in real-time with customer attribution data Build and deploy and price tests to calculate conversion probability and, in turn, optimal premium 3. Every end of month, compare predicted conversion with actual conversion. If outside allowable limits then re-evaluate model automatically

7 Key Take-Aways Ø Ø Ø Insights are hidden in Historical Data on Big Data Platforms Machine Learning and Big Data Analytics find these Insights by building Analytics Models Event Processing uses these Models (without Redevelopment) to take Action in Real Time

8 Agenda 1) Machine Learning and Big Data Analytics 2) Building an Analytic Model 3) Real Time Processing 4) Live Demo

9 Agenda 1) Machine Learning and Big Data Analytics 2) Building an Analytic Model 3) Real Time Processing 4) Live Demo 5) Intelligent Microservices

10 Machine Learning. allows computers to find hidden insights without being explicitly programmed where to look.

11 Real World Examples of Machine Learning Spam Detection Search Results + Product Recommendation Picture Detection (Friends, Locations, Products) The Next Disruption: Google Beats Go Champion Machine Learning is already present in daily life Now, every enterprise is beginning to leverage it!

12 From Insight to Action - Closed Loop for Big Data Analytics Insight EVENTS Action

13 From Insight to Action - Closed Loop for Big Data Analytics ACCESS PREDICT WRANGLE DECIDE Insight Action ANALYZE ACT MODEL MONITOR

Analytics for the Analyst, Data Scientist and Developer AI-DRIVEN VISUAL ANALYTICS DATA DISCOVERY AI-driven visualization to help you

beautiful dashboards in minutes Collaborate with a click DASHBOARDS DATA SCIENCE RE-IMAGINED PREDICTIVE MACHINE LEARNING You don't

Spotfire makes it easy to apply machine learning, big data, and Copyright advanced 2000-2017 analytics TIBCO with Software R, H20, Inc.

14 Analytics for the Analyst, Data Scientist and Developer AI-DRIVEN VISUAL ANALYTICS DATA DISCOVERY AI-driven visualization to help you gain insights at the speed of thought AI-driven data wrangling AI-driven geo analytics Access connected visualizations - build beautiful dashboards in minutes Collaborate with a click DASHBOARDS DATA SCIENCE RE-IMAGINED PREDICTIVE MACHINE LEARNING You don't have to be a data scientist to use data science And Spotfire loves deep data science! Spotfire makes it easy to apply machine learning, big data, and Copyright advanced analytics TIBCO with Software R, H20, Inc. SAS... STREAMING ANALYTICS REAL TIME ACTIONABLE Gain continuous awareness Automate actions and alerts Enable human interaction with live data for better decisions Take informed Action and Win!

15 Analytics Maturity Model Analytics Self-service Visual Analytics Dashboards Advanced Analytics Event Processing Measure Diagnose Predict Optimize Alert Automate Analytics Maturity Immediate Value to the Organization Long-Term Competitive Advantage A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases

16 Analytics Maturity Model Analytics Visual Analytics Advanced Analytics Event Processing Measure Diagnose Predict Optimize Alert Automate Analytics Maturity Immediate Value to the Organization Long-Term Competitive Advantage A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases

17 Analytics Maturity Model Analytics Self-service Visual Analytics Dashboards Advanced Analytics Event Processing Measure Diagnose Predict Optimize Alert Automate Analytics Maturity Immediate Value to the Organization Long-Term Competitive Advantage A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases

18 Insight Platforms

19 The first task in a new analytics projects is to define a Business Case!

Productivity Grow Revenue Reduce Risk Well Production Historian SAP Visualize Clean Shape Merge Enrich

20 From a Business Case to Proactive Actions Business Case Assemble Data Explore Present Develop Model Decision, Action Value Theses Data Wrangling Signals Dashboards Model Prediction Action G&G Increase Productivity Grow Revenue Reduce Risk Well Production Historian SAP Visualize Clean Shape Merge Enrich Filter GeoLocation Completions Production Equipment Pressure Temperature Production Interrupt Equipment Failure Drill Bit Movement

21 Agenda 1) Machine Learning and Big Data Analytics 2) Building an Analytic Model 3) Real Time Processing 4) Live Demo

22 Analytical Pipeline

23 Analytics Maturity Model Analytics Self-service Visual Analytics Dashboards Advanced Analytics Event Processing Measure Diagnose Predict Optimize Alert Automate Analytics Maturity Immediate Value to the Organization Long-Term Competitive Advantage A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases

24 What is Predictive Analytics?

25 Analytical Pipeline

Services Flat Files Siebel ebusiness XML

26 Variety of Data in Enterprises SAP R/3 Oracle E-Business Salesforce Web Services Flat Files Siebel ebusiness XML RDBMS RDBMS RDBMS RDBMS SAP BW OBIEE Spreadsheets D A T A F A B R I C SQL Server Oracle MySQL Teradata Netezza JDBC/ODBC Hadoop SFDC PostgreSQL Etc. Databases Teradata MS SSAS Databases Information Services (join, transform, reusable, parameterized, dynamic query for in-memory use) Direct Query (dynamically query and retrieve data for visualization and analysis) Oracle TeradataAster MySQL Netezza Hadoop Etc. Local data sources Excel Access Drag-and-drop Custom GUI-driven data access via SDK STDF Direct connection ODBC OLE DB SqlClient

27 Data Acquisition Smart Recommendation Engine

28 Analytical Pipeline

29 Data Munging / Wrangling / Mash-up

30 Data Munging - Transformations cust_id dept sku dollar gift date C FALSE A FALSE C TRUE B FALSE C TRUE B FALSE C FALSE n cust_id A B C total # orders first_dat e last_dat e

Feature Engineering The heart of data science You want to get the best accuracy from machine learning algorithms on your datasets Feature engineering is the process of using domain knowledge of the

31 Feature Engineering The heart of data science You want to get the best accuracy from machine learning algorithms on your datasets Feature engineering is the process of using domain knowledge of the data to create features that make machine learning algorithms work. Feature engineering is fundamental to the application of machine learning, and is both difficult and expensive. The process of feature engineering Brainstorming Or Testing features Deciding what features to create Creating features Checking how the features work with your model Improving your features if needed Go back to brainstorming/creating more features until the work is done

32 Analytical Pipeline

33 Exploratory Data Analysis

Exploratory Data Analysis Exploratory Data Analysis (EDA) is an approach/philosophy for data analysis that employs a variety of techniques (mostly graphical) 1.

34 Exploratory Data Analysis Exploratory Data Analysis (EDA) is an approach/philosophy for data analysis that employs a variety of techniques (mostly graphical) 1. to maximize insight into a data set 2. uncover underlying structure 3. extract important variables 4. detect outliers and anomalies 5. test underlying assumptions 6. develop parsimonious models 7. determine optimal factor settings

35 Exploratory Data Analysis The greatest value of a picture is when it forces us to notice what we never expected to see John W. Tukey, 1977

36 Visual Analytics - Interactive Brush-Linked and Inline Data Wrangling à Ad-hoc data preparation instead of just ETL

37 Analytics Maturity Model Analytics Visual Analytics Advanced Analytics Event Processing Measure Diagnose Predict Optimize Alert Automate Analytics Maturity Immediate Value to the Organization Long-Term Competitive Advantage A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases

38 What is Predictive Analytics?

39 Analytical Pipeline

40 Which picture represents a model? A model is a simplification of the truth that helps you with decision making.

41 Model Building

42 Model Building

43 Model Building Employees who write longer s earn higher salaries!

44 Model Improvement

45 Model Improvement Managers Staff

46 Analytical Pipeline

47 Model Validation How is the IQ of a kid related to the IQ of his / her mum?

How to choose the right Machine Learning algorithm? Typical algorithm model selection can be decided broadly on following questions: How much data do you have and is it continuous?

48 How to choose the right Machine Learning algorithm? Typical algorithm model selection can be decided broadly on following questions: How much data do you have and is it continuous? Is it classification or regression problem? Predefined variables (Labeled), unlabeled or mix? Data class skewed? What is the goal? predict or rank? Result interpretation easy or hard?

49 Frameworks and Tooling

50 Smart Data Discovery (for the Business User) as a next-generation data discovery capability that automatically finds and explains insights from advanced analytics to business users or citizen data scientists Leverage Machine Learning without the help of a Data Scientist

51 Advanced Analytics and Big Data Tools (for Data Scientists) Many more.

52 R Language Built for data scientists Very active community

53 R with Revolution Analytics (now Microsoft) Open Source GPL License (including its restrictions)

54 TERR - TIBCO s Enterprise Runtime for R TIBCO has rewritten R as a Commercial Compute Engine Latest statistics scripting engine: S a S-PLUS a R a TERR Runs R code including CRAN packages Engine internals rebuilt from scratch at low-level Redesigned data objects, memory management High performance + Big Data TERR is licensed from TIBCO TERR Installs (free) with Spotfire Analyst / Desktop + other TIBCO products Spotfire Server can manage all TERR / R scripts, artifacts for reuse Standalone Developer Edition Supported by TIBCO No GPL license issues

55 Which R to use?

56 Apache Spark General Data-processing Framework à However, focus is especially on Analytics (at least these days)

57 Apache Spark MLlib Spark ML is Spark s machine learning library. Its goal is to make practical machine learning scalable and easy. x It consists of common learning algorithms and utilities, including classification, regression, clustering and collaborative filtering. General Data-processing Framework à Focus is especially on Analytics (at least these days)

H2O.ai An Extensible Open Source Platform for Analytics Best of Breed Open Source Technology Easy-to-use Web UI and Familiar Interfaces Data Agnostic

58 H2O.ai An Extensible Open Source Platform for Analytics Best of Breed Open Source Technology Easy-to-use Web UI and Familiar Interfaces Data Agnostic Support for all Common Database and File Types Massively Scalable Big Data Analysis Real-time Data Scoring ( Nanofast Scoring Engine )

algorithms Image Detection, Speech Translation, etc.

59 TensorFlow Machine Learning Framework Open sourced by Google Deep Learning à Neural Networks More powerful analysis than other ML algorithms Image Detection, Speech Translation, etc. Computationally very complex, especially during training but also during scoring

60 SaaS Machine Learning Managed SaaS service for building ML models and generating predictions Integrated into the corresponding cloud ecosystem Easy to use, but limited feature set and potential latency issues if combined with external data or applications

61 Live Demo Live Demo Smart Visual Analytics vs. Data Science Tools

62 TIBCO Spotfire with R / TERR Integration Let the business user leverage Analytic Models (created by the Data Scientist) to find insights! Example: Customer Churn with Random Forest Algorithm refresh model button lives a random forest algorithm requires no a priori assumptions at all, it just always works The business user doesn t need to know what random forest is to be empowered by it Select variables for the model

63 TIBCO Spotfire with H2O Integration Example: Predictive Analytics for Manufacturing ( scrap parts as early as possible )

64 TIBCO Spotfire with H2O Integration Example: Predictive Analytics for Manufacturing ( scrap parts as early as possible )

65 PMML (Predictive Model Markup Language ) XML-based de facto standard to represent predictive analytic models Developed by the Data Mining Group (DMG) Easily share models between PMML compliant applications (e.g. between model creation and deployment for operations)

66 Agenda 1) Machine Learning and Big Data Analytics 2) Building an Analytic Model 3) Real Time Processing 4) Live Demo

67 Analytics Maturity Model Analytics Self-service Visual Analytics Dashboards Advanced Analytics Event Processing Measure Diagnose Predict Optimize Alert Automate Analytics Maturity Immediate Value to the Organization Long-Term Competitive Advantage A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases

68 What is Prescriptive Analytics?

69 Traditional Data Processing: Request Response Act Analyze Store

70 The New Era: Streaming Analytics Act & Monitor Store Analyze

71 Streaming Analytics: What Is A Stream? Consists of pieces of data typically generated due to a change of state. One or more identifiers Timestamp & payload Immutable Typically unbounded; there is no end to the data. Batch dataset: bounded. Can be raw or derived. Logs Clickstream Usage Data Sensors

72 Streaming Analytics - Processing Pipeline Applying an Analytic Model is just a piece of the puzzle! Normalization Index / Search Contextual Rules Messaging Filtering Windowing Analytics / DW Reporting APIs Transformation Patterns Analytics Process Management Integration Adapters / Channels Enrichment Aggregation Deep ML Applications & APIs Analytics (Real Time) Stream Ingest Stream Preprocessing Stream Analytics & Processing Stream Outcomes

73 Streaming Analytics: Windows

of history combined with live context and business rules

74 Operational Intelligence and Human Interaction Machine-to-Machine Automation Automated action based on models of history combined with live context and business rules Actions by Operations Human decisions in real time informed by up to date information 74

What Kind of Streaming Analytics do you need? Streaming Concepts Slow Streaming Frameworks Streaming Products Fast Time to Market Library (Java,.

75 What Kind of Streaming Analytics do you need? Streaming Concepts Slow Streaming Frameworks Streaming Products Fast Time to Market Library (Java,.NET, Python) Query Language (often similar to SQL) Scalability (horizontal and vertical, fail over) Connectivity (technologies, markets, products) Operators (Filter, Sort, Aggregate) Visual IDE (Dev, Test, Debug) Simulation (Feed Testing, Test Generation) Live UI (monitoring, proactive interaction) Maturity (24/7 support, consulting) Integration (out-of-the-box: ESB, MDM, etc.)

76 Frameworks and Products (no complete list!) PRODUCT OPEN SOURCE CLOSED SOURCE Azure Microsoft Stream Analytics FRAMEWORK

77 Comparison of Stream Processing Frameworks and Products Slide Deck and Video Recording: streaming-analytics-comparison-open-source-frameworks-products-cloud-services/

78 Apache Storm Hello World

79 AWS Kinesis Hello World

80 Visual Coding for Streaming Analytics Streaming Operators Connectivity Visual Development Testing & Simulation Mature Tooling / Support Middleware Integration

81 Live Visual Analytics UI Ad-hoc continuous query Alerts Dynamic aggregation Action Live visualization

82 TERR Spark ML MATLAB Open Source R SAS How to apply analytic models to real time processing without redevelopment? H20.ai Stream Processing PMML

83 Apache Spark ML and Spark Streaming with PMML Models Not available for Spark 2.0 yet, just for 1.6.x!

84 TIBCO StreamBase Connector for R and TERR

85 TIBCO StreamBase Connector for H2O.ai

86 TIBCO StreamBase Connector for PMML

87 Real World Streaming Application for Customer Churn

88 Closed Loop à Automatically Re-Compute (and Improve) the Analytic Model Compute your performance metric Spot not good enough performance Re-compute model

89 Agenda 1) Machine Learning and Big Data Analytics 2) Building an Analytic Model 3) Real Time Processing 4) Live Demo

90 Scenario: Predictive Scrapping of Parts in an Assembly Line Goal: Scrap parts as early as possible automatically to reduce costs in a manufacturing process. Question: When to scrap a part in Station 1 instead of doing re-work or sending it to Station 2? Station 1 Station 2 Cost Before Total Cost 29 (or more) Scrap? Scrap?

91 Fast Data Architecture for Predictive Maintenance CSV Batch JSON Real Time Action Streaming StreamBase Analytics Aggregate Correlate Analytics Live Datamart Continuous query processing Alerts Operational Analytics XML Real Time Rules Manual action, escalation Live UI Operations Internal Data Flume HDFS HDFS HISTORICAL ANALYSIS R / TERR Data Scientists H2O Oracle RDBMS Avro Parquet PMML Hadoop (Cloudera) Spotfire TIBCO Fast Data Platform

92 TIBCO Spotfire with H2O Integration Data Discovery / Data Mining ( Are parts that repeat a station more likely scrap parts? )

93 TIBCO Live Datamart Operational Intelligence ( Monitor the manufacturing process and change rules in real time! ) Live Dartmart Desktop Client

94 TIBCO Live Datamart Operational Intelligence ( Monitor the manufacturing process and change rules in real time! ) Live Dartmart Web API

95 Live Demo Live Demo TIBCO Spotfire + StreamBase + H2O.ai + Live Datamart

TIBCO Accelerator for Apache Spark The TIBCO Accelerator for Spark is a TIBCO engineered, light-weight open-source faststart for systems to stream data into Spark, discover patterns in Spark with

96 TIBCO Accelerator for Apache Spark The TIBCO Accelerator for Spark is a TIBCO engineered, light-weight open-source faststart for systems to stream data into Spark, discover patterns in Spark with Spotfire, and operationalize the insights on Big Data. FUNCTIONAL COMPONENTS 1. Fast Data Preparation for IoT Dozens of enterprise and IoT data preparation adapters: MQTT, Databases; inbound creation of HDFS, Parquet, Hbase, Avro 2. Spotfire Model Discovery Template Use Spotfire to explore Spark data lake, create predictive model, train in H20, and deploy to Streaming Analytics. 3. Operationalize Predictive Models Zookeeper deployment to StreamBase nodes living in Spark cluster via H20, PMML, TERR models 4. Streaming Analytics for Automation Automate action based on predictive models make offers to customers, stop fraudulent transactions, alert. 5. Monitor & Retrain Model Monitor behavior of model, retrain when necessary. 6. Drag & Drop for Business Solution Developers Code-free development environment for work with H20, HDFS, Avro, TERR

97 Develop fast. Fail fast. Change fast. Visual Analytics + Visual Coding + DevOps = Agile Intelligent Microservices

98 Key Take-Aways Ø Ø Ø Insights are hidden in Historical Data on Big Data Platforms Machine Learning and Big Data Analytics find these Insights by building Analytics Models Event Processing uses these Models (without Redevelopment) to take Action in Real Time

99 Questions? Please contact me! Kai Wähner Technology Evangelist at LinkedIn

Manufacturing Analytics in the Age of IoT

Manufacturing Analytics in the Age of IoT Mike Alperin September 2016 Copyright 2000-2015 TIBCO Software Inc. Speaker Mike Alperin Principal Industry Consultant Data Science Team Copyright 2000-2014 TIBCO