Data Analytics with MATLAB Adam Filion Application Engineer MathWorks

Size: px

Start display at page:

Download "Data Analytics with MATLAB Adam Filion Application Engineer MathWorks"

Joan Richard
6 years ago
Views:

1 Data Analytics with Adam Filion Application Engineer MathWorks 2015 The MathWorks, Inc. 1

2 Case Study: Day-Ahead Load Forecasting Goal: Implement a tool for easy and accurate computation of dayahead system load forecast Requirements: Acquire and clean data from multiple sources Accurate predictive model Easily deploy to production environment 2

3 Challenges with Data Analytics Aggregating data from multiple sources Cleaning data Choosing a model Moving to production 3

4 NYISO Energy Load Data 4

5 Techniques to Handle Missing Data List-wise deletion Unbiased estimates Reduces sample size Implementation options Built in to many functions Manual filtering 5

6 Techniques to Handle Missing Data Substitution replace missing data points with a reasonable approximation Easy to model Too important to exclude 6

7 Merge Different Sets of Data Join along a common axis Popular Joins: Inner Full Outer Left Outer Right Outer Inner Join Full Outer Join Left Outer Join 7

8 Full Outer Join Key A B First Data Set Key X Y Z Second Data Set Key B Y Z NaN 1.4 NaN NaN NaN NaN 0.8 NaN Joined Data Set 8

9 Learn More: Big Data with 9

10 Challenges with Data Analytics Aggregating data from multiple sources Cleaning data Choosing a model Moving to production 10

11 Machine Learning Characteristics and Examples Characteristics Lots of variables System too complex to know the governing equation (e.g., black-box modeling) Examples Pattern recognition (speech, images) Financial algorithms (credit scoring, algo trading) Energy forecasting (load, price) Biology (tumor detection, drug discovery) 11

12 Overview Machine Learning Type of Learning Categories of Algorithms Machine Learning Supervised Learning Develop predictive model based on both input and output data Classification Regression Unsupervised Learning Clustering Group and interpret data based only on input data 12

13 Supervised Learning Regression Neural Networks Decision Trees Ensemble Methods Non-linear Reg. (GLM, Logistic) Linear Regression Classification Support Vector Machines Discriminant Analysis Naive Bayes Nearest Neighbor 13

14 Unsupervised Learning k-means, Fuzzy C-Means Hierarchical Clustering Neural Networks Gaussian Mixture Hidden Markov Model 14

15 Learn More: Machine Learning with mathworks.com/machine-learning 16

16 Challenges with Data Analytics Aggregating data from multiple sources Cleaning data Choosing a model Moving to production 17

17 Deployment Highlights Database Servers Desktop Applications Hadoop / Big Data Spreadsheets Application Servers Client Applications Web Applications Batch/Cron Jobs Share with others who may not have Royalty-free deployment Encryption to protect your intellectual property 18

18 Deploying Applications with Compiler Compiler SDK Standalone Application Excel Add-in Hadoop C/C ++ Java.NET Production Server Compiler for sharing programs without integration programming Compiler SDK provides implementation and platform flexibility for software developers Production Server provides the most efficient development path for secure and scalable web and enterprise applications 19

19 Sharing Standalone Applications Application Author Toolboxes 1 2 Compiler End User Standalone Application Excel Add-in Hadoop 3 Runtime 20

20 Integrating -based Components Application Author Toolboxes Application author and software developer might be same person 1 Software Developer 2 Compiler SDK C/C ++ Java.NET Production Server 3 4 Runtime 21

Production Server Directly deploy analytic programs into production Centrally manage

Scalable & reliable Service large numbers of concurrent requests Add capacity or

application servers Lightweight client library isolates processing Access programs

21 Production Server Directly deploy analytic programs into production Centrally manage multiple programs & MCR versions Automatically deploy updates without server restarts Scalable & reliable Service large numbers of concurrent requests Add capacity or redundancy with additional servers Production Server(s) Use with web, database & application servers Lightweight client library isolates processing Access programs using native data types Integrates with Java,.NET, C and Python Web Server(s) HTML XML Java Script 22

22 Deployed Analytics Production Server Web Application Server Apache Tomcat Production Server Production Server Desktop Train in Web Server/ Webservice Request Broker Predictive Models CTF Weather Data Energy Data 23

23 Learn More: Application Deployment with 24

24 Learn More: Application Deployment Also 25

Data Analytics Products Access and Explore Data Preprocess Data

Computing Toolbox, Distributed Computing Server Production Server

Acquisition Toolbox Curve Fitting Toolbox Neural Network Toolbox

System Toolbox Image Acquisition Toolbox Image Processing Toolbox

25 Data Analytics Products Access and Explore Data Preprocess Data Develop Predictive Models Integrate Analytics with Systems Parallel Computing Toolbox, Distributed Computing Server Production Server Database Toolbox Statistics and Machine Learning Toolbox Compiler Data Acquisition Toolbox Curve Fitting Toolbox Neural Network Toolbox Compiler SDK Mapping Toolbox Signal Processing Toolbox Computer Vision System Toolbox Image Acquisition Toolbox Image Processing Toolbox Econometrics Toolbox Used in today s demo OPC Toolbox Additional Data Analytics products 26

26 Key Takeaways Data preparation can be a big job; leverage built-in tools and spend more time on the analysis Rapidly iterate through different predictive models, and find the one that s best for your application Leverage parallel computing to scale-up your analysis to large datasets Eliminate the need to recode by deploying your algorithms into production 27

27 2015 The MathWorks, Inc. 28

Data Analytics for Engineers

Data Analytics for Engineers Will Wilson Application Engineer MathWorks 2016 The MathWorks, Inc. 1 Agenda Definition Common Challenges Case Study Wrap Up 2 What is Data Analytics? Data Analytics is the