St Louis CMG Boris Zibitsker, PhD

Size: px

Start display at page:

Download "St Louis CMG Boris Zibitsker, PhD"

Justin Higgins
6 years ago
Views:

1 ENTERPRISE PERFORMANCE ASSURANCE BASED ON BIG DATA ANALYTICS St Louis CMG Boris Zibitsker, PhD

2 Abstract Today s fast-paced businesses have to make business decisions in real time. That creates pressure on IT leaders to develop Big Data infrastructure and applications capable to process large volume of data from different sources, apply advanced analytics and present recommendations in real time. These applications work in distributed, multitier, virtualized, parallel processing environment where each cluster and system has own performance management tools and dedicated performance repositories. It creates a lot of obstacles for Applications Performance Management and Capacity Management of Big Data environment. In this presentation we will discuss challenges of creating an Enterprise Performance Assurance platform for processing streams of measurement data in memory using Kafka and Spark and storing filtrated and aggregated data in Data Lake. We will review case studies illustrating application of Big Data advanced analytics, including descriptive, diagnostic, predictive and prescriptive analytics for organizing Enterprise Performance Assurance processes. Optimizing Business and IT

3 Outline Problem Performance Assurance for Big Data World Data Collection Role of Data Lake Data Aggregation and Transformation Application of Advanced Analytics for: Workload Characterization Identification of Seasonal Peaks Workload Forecasting Performance Prediction Performance Management Workload Management Dynamic Capacity Planning Optimizing Business and IT 3

4 PROBLEM Optimizing Business and IT

5 Challenges Each line of business uses different applications, systems and has different SLGs Real time decisions Business transaction often access several systems Complexity Cost Growth Optimizing Business and IT 5

6 Business Transaction Often Processed by Different Systems Clouds Data Centers Systems Hardware Software Subsystems Workloads Applications Data Networks Workloads SLGs Optimizing Business and IT 6

7 Risk of Performance Surprises Software performance engineering POC feasibility study, selection of the platform and infrastructure New application design, development, testing and implementation Applications modification Dynamic capacity management Workload management change of priorities, concurrency and resource allocation Performance management change of OS and Software subsystems parameters, application tuning Capacity planning Hardware upgrade Moving workload from one platform to another Software upgrade Software Performance Engineering Dynamic Capacity Management Capacity Planning Optimizing Business and IT 7

8 SOLUTION Optimizing Business and IT 8

9 Applying Big Data Infrastructure and Advanced Analytics Data Collection Streaming performance measurement data from different systems, subsystems, workloads In memory processing using Kafka and Spark to enable dynamic capacity management Storing aggregated data in Data Lake Use Reservoirs for specific applications Applying Machine learning algorithms for Workload characterization using Descriptive Analytics Workload forecasting and seasonal peaks determination Determining anomalies and their root causes Tuning of OS/Linux, software subsystems and applications Performance prediction Workload management - Priorities and Concurrency Development of recommendations Prescriptive Analytics Optimizing Business and IT 9

10 Multi-Criteria problem of optimization Major Criteria Response Time, Throughput and Cost Major Variables CPU time, I/O, Memory, Network demand by workload Hardware and software configuration Availability - Frequency of errors by workloads and applications and frequency of hardware outages Power consumption Plan Workload and volume of data growth New applications implementation Seasonal peaks Moving workloads between systems Options Hardware, software and Virtual configuration Application tuning Major Limitations Budget SLGs Optimizing Business and IT 10

11 Performance Assurance Technology and Process Technology Big Data Infrastructure Data streaming In memory processing Data storing - Big Data Lakes Workload aggregation and characterization across all systems Electrical Power consumption management Advanced Analytics Descriptive, Diagnostic, Predictive, Prescriptive and Control Applications, Data and Systems life cycle Recommender Automation Process Software Performance Engineering Design and development of new applications for performance Predicting new application implementation impact Dynamic Capacity Management Performance Management Workload Management Long term and short term capacity planning Optimizing Business and IT 11

12 Automation Analytics Human Input Descriptive What happened? Diagnostic Why did it happen? Data Predictive What will happen? Action Decision Action Prescriptive What Should I do? Decision Support Decision Automation Big Data advanced analytics makes possible to implement self-healing, selfadapting systems based on predictive and prescriptive analytics and automate Infrastructure & Operations Management. Source Gartner Group Optimizing Business and IT 12

13 Open Source Solutions AT&T Open Source Solution ECOMP (Enhanced Control, Orchestration, Management & Policy) Architecture White Paper Oracle: Accelerating EPM Deployment With Planning in the Cloud rc2=wwmk mpp013&sc=sckw=wwmk mpp013&mkwid=shwlm GmQb pcrid pkw enterprise%20performance pmt p pdv c sckw=srch:e nterprise%20performance Team Quest CMG webinar on July 20th Optimizing Business and IT 13

14 DATA COLLECTION

15 WORKLOAD AGGREGATION AND CHARACTERIZATION

Agents Data Transformation Workload Aggregation & Characterization Software Performance Engineering Dynamic Capacity Management Capacity Planning Enterprise Performance Assurance Based on Big Data

16 Agents Data Transformation Workload Aggregation & Characterization Software Performance Engineering Dynamic Capacity Management Capacity Planning Enterprise Performance Assurance Based on Big Data Analytics Workload Aggregation and Characterization Process Big Big Data Data Clusters Clusters Teradata, Teradata, Oracle, Oracle, DB2 DB2 EDW EDW Data Lake Other Other IT IT Platforms Platforms Each Workload on Each System has Unique Performance, Resource Utilization and Data Usage Profiles 16

17 Workload Characterization by System Building workloads profiles Performance Resource utilization Data usage Results are used as input for: Workload forecasting Performance management Workload management Capacity planning 17

18 Applying ML Algorithms for Seasonal Peaks Determination WORKLOADNAME PARAMETER PERIOD MEANDURATION MEANAMPLITUDE STD MIN MAX 95PERCENTILE PEAKRANGE PEAKLENGTHSTD ACT TOTALCPUTIME APPLDEV TOTALCPUTIME CAT TOTALCPUTIME CLIENTRPTG TOTALCPUTIME LOAD TOTALCPUTIME FIN TOTALCPUTIME FRAUD TOTALCPUTIME HELPDESK TOTALCPUTIME HR TOTALCPUTIME IT TOTALCPUTIME MKT TOTALCPUTIME DBA TOTALCPUTIME

19 CPU Utilization by Business Workloads During Seasonal Peak Sales Marketing 19

20 PERFORMANCE MANAGEMENT

21 Diagnostics and Root Cause Analysis Anomaly Detection Short term performance prediction using linear regression model If measurement data for an hour is significantly greater than predicted it s as Anomaly Date Time Workload Expected Name Parameter Name Root Cause Real Value Value 1/13/ :00 HR-BATCH MEANRESPTIME /14/2016 9:00 HR-BATCH MEANRESPTIME HR-BATCH MEANIOOPS /14/ :00 HR-BATCH MEANRESPTIME HR-BATCH MEANIOOPS HR-BATCH TOTALEXECCOUNT /15/2016 0:00 HR-BATCH MEANRESPTIME HR-BATCH MEANCPUTIME HR-BATCH MEANIOOPS /15/2016 9:00 HR-BATCH MEANRESPTIME /15/ :00 HR-BATCH MEANRESPTIME HR-BATCH MEANCPUTIME HR-BATCH MEANIOOPS /16/ :00 HR-BATCH MEANRESPTIME /17/2016 3:00 HR-BATCH MEANRESPTIME HR-BATCH MEANIOOPS HR-BATCH TOTALEXECCOUNT /17/ :00 HR-BATCH MEANRESPTIME HR-BATCH TOTALEXECCOUNT /17/ :00 HR-BATCH MEANRESPTIME HR-BATCH MEANIOOPS /17/ :00 HR-BATCH MEANRESPTIME /18/2016 2:00 HR-BATCH MEANRESPTIME HR-BATCH MEANIOOPS /18/2016 6:00 HR-BATCH MEANRESPTIME HR-BATCH MEANIOOPS

22 Anomaly Detection Determining Significant Changes with RT, Throughput and Resource Utilization 22

23 Root Cause Determination For workload having response time anomaly check if throughput, CPU time and number of I/O operations have an anomaly and find users and programs responsible for that Check if other workloads, users and programs had throughput, CPU time and number of I/O operations anomaly at the same time DateTime WorkloadCause ProgramName UserName 1/14/2016 9:00 CIIOUT-BATCH MEANIOOPS None CII_SL_NO_DSHBRD_OUT 1/14/ :00 CIIOUT-BATCH MEANIOOPS JOBSERVERCHILD CII_SL_NO_DSHBRD_OUT 1/14/ :00 CIIOUT-BATCH TOTALEXECCOUNT JOBSERVERCHILD CII_SL_NO_DSHBRD_OUT 1/15/2016 0:00 CIIOUT-BATCH MEANCPUTIME JOBSERVERCHILD CII_SL_ALL_RPT_OUT 1/15/2016 0:00 CIIOUT-BATCH MEANIOOPS JOBSERVERCHILD CII_SL_ALL_RPT_OUT 1/15/ :00 CIIOUT-BATCH MEANCPUTIME CRPROC CII_SL_ALL_RPT_OUT 1/15/ :00 CIIOUT-BATCH MEANIOOPS CRPROC CII_SL_ALL_RPT_OUT 1/17/2016 3:00 CIIOUT-BATCH MEANIOOPS JOBSERVERCHILD CII_SL_NO_DSHBRD_OUT 1/17/2016 3:00 CIIOUT-BATCH TOTALEXECCOUNT JOBSERVERCHILD CII_SL_NO_DSHBRD_OUT 1/17/ :00 CIIOUT-BATCH TOTALEXECCOUNT JOBSERVERCHILD CII_SL_NO_DSHBRD_OUT 1/17/ :00 CIIOUT-BATCH MEANIOOPS None CII_SL_NO_DSHBRD_OUT 1/18/2016 2:00 CIIOUT-BATCH MEANIOOPS JOBSERVERCHILD CII_SL_NO_DSHBRD_OUT 1/18/2016 6:00 CIIOUT-BATCH MEANIOOPS None CII_SL_ALL_RPT_OUT 23

24 Root Cause Determination Algorithms Decision Trees Logistic Regression Analysis Predictive analytics Decision Tree - Leaf page and branches identify the root cause 24

25 DYNAMIC CAPACITY MANAGEMENT

26 Short Term Prediction Variable Learning Interval (hours) Prediction Interval (hours) Response Time Throughput CPU Time # I/Os

27 Workload Management Priorities Concurrency Resource Allocation

28 Examples of Workload Management by Queues By Organization 100% Marketing 33% Finance 33% Sales 33% Marketing (33%) By Organization (100%) Finance (33%) Sales (33%) By Type of Workload 100% Near Real Time 70% Batch 30% By Type of Workload (100%) Near Real Time (70%) Batch (30%) Hybrid 100% Marketing 20% Batch 15% Real-Time 5% Finance 40% Real-Time 10% Batch 30% Marketing (20%) Batch (15%) Real Time (5%) Hybrid (100%) Finance (40%) Batch (30%) Real Time (10%) Sales (Batch 40%) Sales Batch 40% 28

29 Examples of Resource Manager Scheduler FIFO Scheduler Processing Jobs in order Capacity Scheduler (Default) Queue shares as percentage of clusters FIFO scheduling within each queue Supporting preemption Fair Scheduler Fair to all users 29

30 Example of Capacity Scheduler Marketing Finance Sales Queue 3 Queue 2 Queue 1 Guaranteed Resources 30% 50% 20% Set Limits on Capacity Minimum capacity for the queue Maximum capacity (% of cluster resources) for a queue Resource elasticity when not being used by other queues Minimum User Limits user sharing for a given queue User Limit factor Maximum queue capacity that one user can take up Application Limit Maximum # of applications submitted to one queue 30

31 Example of Predicting Workload Concurrency Change Impact 31

32 Example of Predicting Workload Priority Change Impact 32

33 CAPACITY PLANNING 33

34 Input and Output for Performance Prediction and Dynamic Capacity Management and Capacity Planning 34

Long Term Prediction Predicting Workload Growth Impact Apply Predictive Analytics Predict the impact of expected workload and volume of data

upgrade will affect performance of the individual workloads Apply Prescriptive Analytics Evaluate options Justify what should be done

35 Long Term Prediction Predicting Workload Growth Impact Apply Predictive Analytics Predict the impact of expected workload and volume of data growth Predict how new application will perform on production system Individual Hadoop Clusters vs Data Lake Determine how planned hardware upgrade will affect performance of the individual workloads Apply Prescriptive Analytics Evaluate options Justify what should be done proactively to meet SLGs Set realistic expectations Predict the impact of workload and volume of data growth Determine when workloads SLGs will not be met 35

Predicting New Application Implementation

Application More Data More Users Capacity

36 Predicting New Application Implementation Impact Test New GB Data Collection Workload Characterization Workload Forecasting Modeling Test and Production Systems Predicting new Application Implementation Impact Verification New Application More Data More Users Capacity Demand Production Current Applications: Sales, Mkt, HR, ERP, ETL PB 36

Predicting Impact of Different Changes and Justification of Decision Technique: queueing theory, machine learning, data mining, analytic and simulation modelling and game theory.

37 Predicting Impact of Different Changes and Justification of Decision Technique: queueing theory, machine learning, data mining, analytic and simulation modelling and game theory. Example: predicting the impact of the expected increase in number of users and volume of data, implementation of new applications Comparison of different options hardware upgrades, server, data and application consolidation, virtualization, moving workloads between systems Justification of decision Predict how new application will affect performance of existing applications Predict the impact of hardware upgrade 37

38 PRESCRIPTIONS 38

Prescriptive Analytics - Advice Prescriptive Analytics Use outputs of Descriptive, Diagnostic and Predictive Analytics to Recommend: what and when should be done to most economically and effectively

39 Prescriptive Analytics - Advice Prescriptive Analytics Use outputs of Descriptive, Diagnostic and Predictive Analytics to Recommend: what and when should be done to most economically and effectively achieve business goals and meet SLAs. Technique machine learning, artificial intelligence, queueing theory, and optimization algorithms, compare impact of different decision options Value Better-informed decisions Reduce risk Set realistic expectations. Enables Verification and Automation Evaluates prediction results to find how most effectively satisfy SLGs of each workload Recommends Tuning, Workload Management and Capacity Planning actions to continuously meet SLGs 39

40 VERIFICATION AND AUTOMATION

41 Verification Actual vs. Expected (A2E) is a base for feedback control 41

Goal is Automation Data Collection 24*7 Configuration Auto Discovery Workload Characterization Determining Seasonal Peaks Modeling and Performance Prediction Setting Rules Automating Resource

42 Goal is Automation Data Collection 24*7 Configuration Auto Discovery Workload Characterization Determining Seasonal Peaks Modeling and Performance Prediction Setting Rules Automating Resource Allocation and Workload Management Dynamic change of rules based on short term predictions Verification A2E New Prescription Control Optimization of Infrastructure and organization of the continuous proactive management process 42

43 SUMMARY

44 Value of Performance Assurance Unification of Performance Assurance methodology and tools across multiple platforms Optimization of Design, Development and Testing Optimization of Dynamic Capacity Management - Performance Management and Workload Management Optimization of Capacity Planning Optimization of Big Data Infrastructure Setting Realistic Expectations Enables Verification Automation of Performance Assurance process Reduce uncertainty and risk of performance surprises Collaborative capacity management process 44

45 QUESTIONS? 45

Data Transformation Workload Aggregation Enterprise Performance Assurance Based on Big Data Analytics Collection and Aggregation of Performance Measurement Data Agent Manager Big Big Data Data

46 Data Transformation Workload Aggregation Enterprise Performance Assurance Based on Big Data Analytics Collection and Aggregation of Performance Measurement Data Agent Manager Big Big Data Data Clusters Clusters Teradata, Teradata, Oracle, Oracle, DB2 DB2 EDW EDW Other Other IT IT Platforms Platforms Auto Discovery Agent Linux Agent Kafka Agent Spark Agent Storm Agent Cassandra Agent YARN Agent Tez Agent Other Agents Data Lake Performance DW PERFORMANCE Assurance Workload Characterization Workload Forecasting Performance Prediction Workload Management Performance Management Capacity Planning Verification & Control ADVANCED Analytics Descriptive Analytics Diagnostic Analytics Predictive Analytics Prescriptive Analytics Control Analytics 46

MapR Pentaho Business Solutions

MapR Pentaho Business Solutions The Benefits of a Converged Platform to Big Data Integration Tom Scurlock Director, WW Alliances and Partners, MapR Key Takeaways 1. We focus on business values and business