Real Time Analytics. Our journey towards a real time data platform. June 7, 2017 F.H.A. (Floran) Stuijt MSc Alliander IT Data & Insights

Transcription

1 Real Time Analytics Our journey towards a real time data platform June 7, 2017 F.H.A. (Floran) Stuijt MSc Alliander IT Data & Insights

2 Personal Introduction Background Artificial Intelligence Computer Science Previous life Software & Systems Architecture/Engineering (Recording, Robotics) Interests Analytics,(Agile) systems architecture, cognitive models Floran Stuijt MSc IT Alliander floran.stuijt@alliander.com linkedin.com/in/floran-stuijt 2

3 Alliander & Liander 3,0 million customers 5,7 million connections employees 12 billion assets km electricity grid km gas grid 1,7 billion revenues Datum Titel van de presentatie 3

4 Becoming a data driven business Using smart technology data analysis And by closely monitoring the impact of the energy transition We know our grid s status, allowing us to Prevent outages Fix outage faster Focused investments Datum Titel van de presentatie 4

5 operational strategic A Selection of Use Cases Impact of Energy Transition Condition based modeling Data quality monitoring Fault Location Isolation and Supply Restauration (FLISR) Real-time customer notification (status updates) Dynamic load balancing Monitoring and surveillance (e.g. meter alarms) streaming batch 5

6 First: a classification of data Transactional Geographic Real-time 6

7 Transactional Data Planned Work and Maintenance Asset (life cycle) Information Human Resource Information Transactional 7

8 Geographic Data Asset Geo Coding - Asset coordinates Grid Topology - Connectivity between assets Traveling distance Geographic 8

9 Real-time Data EMS/DMS Alerts & Metrics Digital Fault Indicator Alerts Social Media Events Power Quality (voltage threshold exceedance) Meteorological Events (e.g. lightning discharge) Smart meter power on/off notifications Real-time 9

10 Becoming a Data Driven Business requires Integration Transactional Geographic BI&A Real-time Business Intelligence & Analytics is at the heart of integration 10

11 Real-time How to deal with different dynamics of data?

12 Our idea of real-time analytics External Service Filtering outliers and corrupt data Reduce, normalize, aggregate, etc. Enrichment Combining Publishing Key Filter Pipe Eventsource Eventreceiver Call to external service 12

13 Build on top of abstractions Externe Service B Filteren uitschieters en corrupte data Reduceren, normaliseren, aggregeren etc. Verrijken 13

14 Architecture Event Stream Processor Basic Enrich Transform Aggregate Combine Advanced Detect (spatio-temporal) patterns Parallelize (branch & merge) Consult external systems 14

15 Architecture Message Queue Event Stream Processor Decouple source and ESP Act as buffer in-between 15

16 Architecture Message Queue Event Stream Processor Measurements & Event Repository Store raw and derived events for later analysis Distill rules for ESP 16

17 Architecture Real-time Data Platform (RTDP) Message Queue Event Stream Processor Measurements & Event Repository Smart Data Streaming 17

18 Real-time Data Platform Real-time Analytics + Analytics of Real-time Real-Time Data Platform 18

19 Some Challenges Event data modeling A relatively new paradigm Privacy & Security Estimating (initial) scale of the platform Which processing to do where (in ESP or outside) Integrating with Kafka Probably more to come 19

20 Use Case: Fault Location with IntellEvent 20

21 Traditional Grid Operator Tasks (domain) knowledge & experience Customer Calls EMS Alerts Grid Topology Digital Fault Detector Alarms 21

22 IntellEvent Approach Digital Fault Detector Alarms Customer Calls EMS Alerts Asset Condition Model IntellEvent Faster and more adequate response to outage Grid Topology 22

23 IntellEvent Screenshot 23

24 Key takeaways Start experimenting, a lot can be done using open source Make sure business is involved early because of paradigm shift Be aware of the associated challenges Be prepared for an explosion of ideas by introducing this capability 24

25 Any Questions? 25