Extracting business value from Big Data. Dr. Rosaria Silipo Phil Winters

Transcription

1 Extracting business value from Big Data Dr. Rosaria Silipo Phil Winters

2 At Last years KNIME UGM Dr. Killian Thiel Dr. Tobias Kötter TEXT MINING MEETS NETWORK MINING 2

3 White Papers and complete workflows available on the KNIME Public Server! Text Mining for Sentiment Network Mining for Relevance Drill Down on special cases Analytics for Prediction 3

4 4

5 Hot Topics Telemetry Data Manipulation Time Series Analysis Combine with Predictive Analytics SENSIBLE usages of Big Data Measurable / Applied to the Business Use public data please so we can all learn 5

6 Industries with these challenges: Manufacturing Chemical Life Science Transportation Utilities Automotive Cyber Security 6

7 Energy Industry Complex Networks Regulation Green Initiatives Competition Smart Meters / Transmission 7

8 The Irish Energy Trials Smart Meters 1 Year Trial Electricity and Gas 5000 homes and businesses Full before/after Surveys 176 million Rows, 40 G of Lovely Data.. 8

9 Overview Import + Transform Data Clustering of Meter IDs Adding the Survey Results Analysing the Clusters Forecasting Electricity Usage Forecasting Peaks DataRush For KNIME Conclusions and next steps

10 Import Data 1-2

11 Clustering Meter IDs Average hourly time series cluster by cluster 30 clusters with k-means on average daily, monthly, hourly,... kw values

12 Almost Night Owls Avg daily kw Avg hourly kw size Cluster % 7% 17% 18% 20% 73 Cluster % 6% 13% 15% 13% 513 Cluster % 5% 16% 18% 25% 684 Cluster % 7% 17% 18% 24% 230 Cluster % 6% 19% 18% 20% 622 Cluster % 7% 20% 21% 19% 18 Cluster % 6% 15% 18% 29% People in these clusters use electricity more during the night than during the day 2. Cluster 10 uses really a lot of electricity all day round (Fridges? Machines?)

13 Night Owls Avg daily kw Avg hourly kw size Cluster % 7% 9% 9% 14% 58 Cluster % 7% 12% 13% 14% 59 Cluster % 7% 13% 15% 18% 30 Cluster % 7% 14% 14% 15% 439 Cluster % 6% 13% 15% 24% People in these clusters use electrcity mostly at night

14 All Rounders Avg daily kw Avg hourly kw size Cluster % 7% 21% 22% 20% 40 Cluster % 7% 15% 17% 33% 539 Cluster % 6% 20% 21% 24% 507 Cluster % 14% 18% 15% 18% 231 Cluster % 6% 25% 25% 17% 71 Cluster % 5% 18% 24% 19% 418 Cluster % 6% 21% 21% 29% 365 Cluster % 6% 29% 22% 29% 273 Cluster % 6% 29% 27% 16% These people use electricity all day round 2. Besides cluster 14 and cluster 3, they all use little electricity distributed during the day

15 Night and Day Avg daily kw Avg hourly kw size Cluster % 6% 28% 29% 19% 17 Cluster % 5% 35% 34% 15% 25 Cluster % 5% 35% 34% 15% 71 Cluster % 2% 37% 39% 14% More people use electricity at night than during the day 2. Cluster 8 uses a lot of electricity and mostly during the day 3. Cluster 9 uses very little electricity and mostly during the day

16 Survey Data available 3-4 Allocation - type if billing Residential - pre/post Survey housing type, size, age, people, insulation, attitudes, etc. SME (small and medium Enterprises) - pre /post Survey building type, size, age, industry, attitudes, etc.

17 Survey Data Survey Data Decision Tree: Time Series Cluster Predicts the Customer Type! Other Significant Fields: Industry Size in sq. m. Age of building Invested in Efficiency Owner/Renter! Meter ID and Clusters

18 Night and Day Avg daily kw Avg hourly kw size Cluster % 6% 28% 29% 19% 17 Cluster % 5% 35% 34% 15% 25 Cluster % 5% 35% 34% 15% 71 Cluster % 2% 37% 39% 14% More people use electricity at night than during the day 2. Cluster 8 uses a lot of electricity and mostly during the day 3. Cluster 9 uses very little electricity and mostly during the day

19 kw vs. hour of day cluster 8 kw Hour of day

20 Low electricity usage Cluster 8 Christmas 2009 Christmas 2010 kw Hour of day

21 On a smaller time window... Jul Weekly Periodicity Aug kw Monday Jul-20 Monday Jul-27 Tuesday Monday Aug-04 Aug-03 Bank Holiday Monday Aug-10 Monday Aug-17 Hour of day

22 On a smaller time window... Jul Weekly Periodicity Aug kw Monday Jul-20 Monday Jul-27 Monday Aug-03 Bank Holiday Hour of day

23 On a smaller time window... Wed Nov Daily Periodicity Thu Nov kw 11:00 14:00 11:00 14:00 21:00 5:00 21:00 5:00 Hour of day

24 Seasonality 12:00 12:00 16:00 5:00 Wed Jul :00 5:00 Thu Jul :00 11:00 14:00 21:00 5:00 21:00 5:00 Wed Nov Thu Nov

25 Energy and Forecasting Accuracy: What it means An improvement in forecasting accuracy of 1% was estimated to yield a saving in operating costs of approximately 10 million per year Bunn and Farmer 1984 Daily: Optimal Scheduling, Allocation Weekly: Purchase Policies, Maintenance Monthly, Yearly: Strategic Planning and Production

26 Simple Auto-Regressive Model Lag = p Transpose data from x(t) to x(t-p)... x(t-1) x(t) Correlation between x(t) and its past? Linear / Polynomial Regression of x(t) on its past MSE on test set

27 Building x(t-p),..., x(t-1), x(t) lag = p = 3 From QuickForm in Metanode

28 Auto-correlations Lag = 24 hours 24 hours Seasonality

29 Linear/Polynomial Regression Regression Target = x(t) Mean Square Error 90% training set 10% test set

30 Prediction Example Prediction from the Linear Regression:

31 Seasonality 1-24h We need a 24h template to repeat: First 24h Average 24h on training set Previous 24h Create and remove template from signal x(t+i) = x(t+i) template(i) Add template back into predictions p(t+i) = p(t+i) + template(i) Simple AR Model

32 After Seasonality Extraction Range = [-23, 7]

33 Seasonality 1 24h We need a 24h template to repeat: First 24h Average 24h on training set Previous 24h Remove template from signal x(t) = x(t) x(t-24) Add template back into predictions p(t) = p(t) + x(t-24) Simple AR Model

34 After Seasonality Extraction Range = [-19, 24]

35 Seasonality 2 24h * 7 Remove template from signal x(t) = x(t) x(t-24*7) Simple AR Model Add template back into predictions p(t) = p(t) + x(t-24*7)

36 After Seasonality Extraction Range = [-5, 7]

37 Prediction Example

38 Neural Networks Lag = p Remove template from signal x(t) = x(t) x(t-24*7) Add template back into predictions p(t) = p(t) + x(t-24*7) Data back to original range Input for NN Model must be in [0,1] NN Model MSE

39 R arima(p,d,q) R arima model including seasonality MSE

40 Regression and NN Results MSE on cluster 8 test set: MSE / Lag Seasonality Linear AR Polynomial* AR Linear AR First 24h Linear AR Previous 24h Linear AR 24h * NN 24h * *Polynomial Regression with degree = 3

41 R arima Results MSE on cluster 8 test set: MSE / (p, d, q) Seasonalit y (1, 0, 0) (2, 0, 0) (1, 0, 1) (2, 0, 1) (2, 0, 2) R arima(p,0,q) Previous 24h* *arima(gtemp, order=c(p,0,q), xreg=1:nobs, seasonal=list(order=c(p,0,q),period=24), include.mean=true) Note. The R arima -> optim procedure was failing for p > 2 or q > 2, due to memory problems.

42 R arima(1, 0, 1) Predictions

43 MSE on Peaks only MSE on cluster 8 test set only on peaks (pred(t) OR x(t) > 35 kw): MSE / Lag Seasonalit y Linear AR 24h * NN 24h *

44 Big Data

45 Import + Aggregate Data (KNIME) Several hours execution time Reading takes only 20 minutes 7h execution time: 2h datetime conversions 5h Sorter node Several hours execution time

46 Import + Aggregate Data (DataRush) 9 Minutes

47 Big Data Forecasting Clusters Posibilities: More Data or Forecasting over Meter IDs

48 Big Data Conclusions Accessing Data: Manipulating Data: Explore the Data: Predictive Mining: Execution: Real Time: Benificial Very Likely Beneficial Don t Bother Task Based! Experiment Possibly Very Likely Beneficial With KNIME and Pervasive s Rushanalyticsfor KNIME, You can mix and match as required!!!

49 Next Steps: Data Science Introduce the % weekday usage feature New clusters including % weekday usage Better R arima model Investigate more interesting clusters (cluster28 and cluster 2 of the night owls, and cluster 9 of the morning people) Introduce MA(q) model for arima(p,d,q) Implement automatic detection of p, d, and q of arima(p,d,q)

50 Next Steps: Business More Accurate Weekly, Monthly, Yearly Forecasting Pricing plans based on segments Models that relate key golden questions to predicted usage patterns We can predict the meter id s cluster! Using: Customer Type Industry Owner/Renter! Size in sq. m. Age of building Invested in Efficiency

51 Conclusion Hot Topics Telemetry Data Manipulation Time Series Analysis Combine with Predictive Analytics SENSIBLE usages of Big Data Measurable / Applied to the Business 51

52 Industries can now rock the Big Data Challenges! Manufacturing Chemical Life Science Transportation Utilities Automotive Cyber Security Slides will be available, White Paper coming! 52