Oracle Stream Analytics

Size: px

Start display at page:

Download "Oracle Stream Analytics"

Lizbeth Patrick
6 years ago
Views:

1 Oracle Stream Analytics Ereignisverarbeitung leicht(er) gemacht! Guido Schmutz BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

com Blog: http://guidoschmutz.wordpress.

2 Guido Schmutz Working for Trivadis for more than 19 years Oracle ACE Director for Fusion Middleware and SOA Co-Author of different books Consultant, Trainer, Software Architect for Java, SOA & Big Data / Fast Data Member of Trivadis Architecture Board Technology Trivadis More than 25 years of software development experience Contact: guido.schmutz@trivadis.com Blog: Slideshare: Twitter: gschmutz 2

Erbringung von IT-Services mit Fokussierung auf - und -Technologien in der Schweiz, Deutschland,

3 Unser Unternehmen. Trivadis ist führend bei der IT-Beratung, der Systemintegration, dem Solution Engineering und der Erbringung von IT-Services mit Fokussierung auf - und -Technologien in der Schweiz, Deutschland, Österreich und Dänemark. Trivadis erbringt ihre Leistungen aus den strategischen Geschäftsfeldern: B E T R I E B Trivadis Services übernimmt den korrespondierenden Betrieb Ihrer IT Systeme. 3

Mit über 600 IT- und Fachexperten bei Ihnen vor Ort. KOPENHAGEN HAMBURG 14 Trivadis Niederlassungen mit über 600 Mitarbeitenden. Über 200 Service Level Agreements. Mehr als 4'000 Trainingsteilnehmer.

4 Mit über 600 IT- und Fachexperten bei Ihnen vor Ort. KOPENHAGEN HAMBURG 14 Trivadis Niederlassungen mit über 600 Mitarbeitenden. Über 200 Service Level Agreements. Mehr als 4'000 Trainingsteilnehmer. DÜSSELDORF Forschungs- und Entwicklungsbudget: CHF 5.0 Mio. FRANKFURT Finanziell unabhängig und nachhaltig profitabel. GENF BASEL BERN LAUSANNE FREIBURG BRUGG ZÜRICH STUTTGART MÜNCHEN WIEN Erfahrung aus mehr als 1'900 Projekten pro Jahr bei über 800 Kunden. 4

5 Agenda 1. Introduction to Streaming Analytics 2. Oracle Stream Analytics 3. Demo 5

6 Introduction to Streaming Analytics 6

7 Traditional Data Processing - Challenges Introduces too much decision latency Responses are delivered after the fact Maximum value of the identified situation is lost Decision are made on old and stale data Data a Rest 7

8 The New Era: Streaming Data Analytics / Fast Data Events are analyzed and processed in real-time as the arrive Decisions are timely, contextual and based on fresh data Decision latency is eliminated Data in motion 8

9 Event / Stream Processing Architecture Data Sources Data Ingestion (Analytical) Real-Time Data Processing Result Store Data Consumer ERP RDBMS Reports Logfiles Content Social Machine Channel Stream/Event Batch Processing compute Result Store Messaging Service Analytic Tools Alerting Tools Sensor = Data in Motion = Data at Rest Architektur von Big Data Lösungen

10 Lambda Architecture for Big Data Data Sources Logfiles ERP RDBMS Data Ingestion (Analytical) Batch Data Processing Pulling Ingestion Raw Data (Reservoir) Batch compute Computed Information Result Store Result Store Query Engine Data Consumer Reports Service Content Social Channel (Analytical) Real-Time Batch Data Processing compute Analytic Tools Machine Stream/Event Processing Result Store Alerting Tools Sensor Messaging = Data in Motion = Data at Rest Architektur von Big Data Lösungen

11 When to Stream / When not? Constant low Milliseconds & under Low milliseconds to seconds, delay in case of failures 10s of seconds of more, Re-run in case of failures Real-Time Near-Real-Time Batch 11

12 No free lunch Constant low Milliseconds & under Low milliseconds to seconds, delay in case of failures 10s of seconds of more, Re-run in case of failures Real-Time Near-Real-Time Batch Difficult architectures, lower latency Easier architectures, higher latency 12

13 Why Event / Stream Processing? Visualize Business in real-time Dashboards can help people to visualize, monitor and make sense of massive amount of incoming data in real-time Detect Urgent Situations Based on simple or complex analytical patterns of urgent business events Urgent because they happen in real-time Automate immediate actions Run in the background quietly until detecting an urgent situation (risk or opportunity) Alerts can go to humans through , text or push notifications or to other applications trough message queues or service call 13

14 Streaming analytics is anything but a sleepy, rear view mirror analysis of data.

15 15

16 Oracle Stream Analytics 16

2015 Oracle Stream Explorer (SX) Oracle Event Processing for Java Embedded Oracle

17 History of Oracle Stream Analytics 2007 BEA Weblogic Event Server Oracle CQL 2008 Oracle Complex Event Processing (OCEP) 2012 Oracle Event Processing (OEP) Oracle Stream Explorer (SX) Oracle Event Processing for Java Embedded Oracle IoT Cloud Service Oracle Edge Analytics (OAE) 2016 Oracle Stream Analytics (OSA) 17

Oracle Stream Analytics: From Noise to Value Computing Edge Enterprise OEA FOG Devices / Gateways EDGE Analytics Filtering Correlation Aggregation Pattern

Temporal Processing Pattern Matching Machine Learning High Volume Continuous Streaming Sub-Millisecond Latency Disparate Sources Time-Window Processing

18 Oracle Stream Analytics: From Noise to Value Computing Edge Enterprise OEA FOG Devices / Gateways EDGE Analytics Filtering Correlation Aggregation Pattern matching Macro-event High-value Actionable In-context Services Stream Analytics High Volume Continuous Streaming Extreme Low Latency Disparate Sources Temporal Processing Pattern Matching Machine Learning High Volume Continuous Streaming Sub-Millisecond Latency Disparate Sources Time-Window Processing Pattern Matching High Availability / Scalability Coherence Integration Geospatial, Geofencing Big Data Integration Business Event Visualization Sea of data Action! 18

Oracle Stream Analytics Platform What it does Compelling, friendly and visually stunning real time streaming analytics user experience for Business users to dynamically create and implement Instant

19 Oracle Stream Analytics Platform What it does Compelling, friendly and visually stunning real time streaming analytics user experience for Business users to dynamically create and implement Instant Insight solutions Key Features Analyze simulated or live data feeds to determine event patterns, correlation, aggregation & filtering Pattern library for industry specific solutions Streams, References, Maps & Explorations Benefits Accelerated delivery time Hides all challenges & complexities of underlying real-time event-driven infrastructure 19

20 Oracle Stream Analytics Self-Service Stream Processing! Understanding of CQL Filtering, Correlation, Pattern: NOT NEEDED Understanding of IT Deployment and Management: NOT NEEDED Understanding of Development, Java, Best Practices: NOT NEEDED Understanding of the Event Driven Platform: NOT NEEDED 20

21 Oracle Stream Analytics Terminology Explorer: The Application User Interface Catalog: The repository for browsing resources 21

22 Oracle Stream Analytics Terminology Stream: incoming flow of events that you want to analyze (CSV, Kafka, JMS, Rest, MQTT, ) Exploration: application that correlates events from streams and data sources, using filters, groupings, summaries, ranges, and more 22

How the business data is represented in the selected stream Reference: A

23 Oracle Stream Analytics Terminology Shape: A blueprint of an event in a stream or data in a data source. How the business data is represented in the selected stream Reference: A connection to static data that is joined to a stream to enrich it and/or to be used in business logic and output Map: collection of geo-fences 23

24 Oracle Stream Analytics Terminology Pattern: A pre-built Exploration that addresses a particular business scenario in a focused and simplified User Interface Connection: collection of metadata required to connect to an external system Targets: defines an interface with a downstream system 24

Business accessibility to Geo-Streaming Analytics Real Time Streaming Solutions face an increasing need to track "assets of interest" and initiate actions based

25 Business accessibility to Geo-Streaming Analytics Real Time Streaming Solutions face an increasing need to track "assets of interest" and initiate actions based on encroachment of boundary proximity to fixed and moving objects and other geographic, temporal, or event conditions. Geo-Streaming Geo-Fence, Fence, Polygon 25

including mathematical, statistical analysis to the live output

26 Expression Builder enabling calculations Add value to your real time streaming data discovery and analytics by applying and including mathematical, statistical analysis to the live output stream These streaming Excel spreadsheets really do come to life 26

27 Concept of Connections and their reuse in Streams 27

28 Decision Table for Nested IF-THEN-ELSE Rules 28

29 Topology View and Navigation 29

30 Relationship between Streams (Sources), References and Explorations 30

31 Demo 31

Oracle Stream Analytics Demo Use Case: Truck

JSON Geo-Fencing NEAR ENTER Dashboard 2016-06-02

605 98 27 Mark Lochbihler 803014426 Wichita to

21 5187297736652502631 {"timestamp": "2016-06-02

"drivername": "Rommel Garcia", "routeid":

Hanibal", "eventtype": "Normal", "latitude": 37.

32 Oracle Stream Analytics Demo Use Case: Truck Movements Truck Movement Data Ingestion Movement JSON Geo-Fencing NEAR ENTER Dashboard :39: Mark Lochbihler Wichita to Little Rock Route 2 Normal {"timestamp": " :39:56.991", "truckid": 99, "driverid": 31, "drivername": "Rommel Garcia", "routeid": , "routename": "Springfield to KC Via Hanibal", "eventtype": "Normal", "latitude": 37.16, "longitude": "-94.46", "correlationid": } Truck Driver Reckless Driving Detector Reckless Driver 32

Continuous Ingestion in Stream Processing File Source 33 Log

Native REST Dataflow GW Log Topic CDC GW CDC Dataflow GW

Topic Topic Topic Topic Topic Topic Big Data Stream Processing

33 Continuous Ingestion in Stream Processing File Source 33 Log 33 Log Log DB Source CDC DB Source Log Social IoT Sensor Native REST Dataflow GW Log Topic CDC GW CDC Dataflow GW Dataflow MQTT GW Topic Native Connect REST Event Hub Topic Topic Topic Topic Topic Topic Topic Big Data Stream Processing IoT Sensor IoT Sensor IoT GW Queue Architektur von Big Data Lösungen

streams, ) Producer Producer Producer Kafka Cluster Consumer Consumer Consumer Topic Semantic does not

34 Apache Kafka High-volume messaging system Distributed publish-subscribe messaging system Designed for processing of high-volume, real time activity stream data (logs, metrics collections, social media streams, ) Producer Producer Producer Kafka Cluster Consumer Consumer Consumer Topic Semantic does not implement JMS standard! Initially developed at LinkedIn, now part of Apache 34 Internet of Things (IoT) and Big Data

35 Demo: Oracle Stream Analytics 35

36 Demo: Oracle Stream Analytics 36

37 Demo: Oracle Stream Analytics 37

38 Demo: Oracle Stream Analytics 38

39 Summary 39

40 Native Stream Processing => OEP server Event Source Individual Event Event Source Ingestion Event Source P P P P P P P P P P P P 40

41 Micro-Batch Stream Processing => Spark Streaming Event Source Event Source Ingestion Event Source P P P P P P 41

42 Summary Oracle Stream Analytics leverages the capabilities found in Oracle Event Processing (OEP) Empowering Business users to gain insight into real-time information and take appropriate actions when needed => makes stream processing accessible Makes Stream/Event Processing less technical => Excel spread sheet on Streams Part of Oracle IoT Cloud Service Support Spark Streaming as a deployment platform for Streaming ML Interesting road map: Rule Engine, Machine Learning, Extensible Patterns 42

Big Data, IoT & Data Science Internet of Things Device & Gateway Management Analytics at the Edge All the rest of Big Data Advanced Analytics Data Mining / Predictive Analytics Semantic Web

43 Big Data, IoT & Data Science Internet of Things Device & Gateway Management Analytics at the Edge All the rest of Big Data Advanced Analytics Data Mining / Predictive Analytics Semantic Web Visualization Big I Data I Warehouse Convergence BI & Big Data LDW Logical Data Warehouse DWH Archive Unified Query (RDBMS ó Big Data) Big Data Consulting & Managed Services Large & Speedy Data Also known as Big & Fast Hadoop Ecosystem NoSQL DBs Event Hubs & Streaming Analytics Data Lake Big Data, IoT & Data Scientist Trainings 43

Oracle Stream Analytics on Docker Oracle Stream Analytics 12.

44 Oracle Stream Analytics on Docker Oracle Stream Analytics Documentation Oracle Stream Analytics Download 44

45 Guido Schmutz Technology Manager 45

Internet of Things (IoT) Are traditional architectures good enough?

Internet of Things (IoT) Are traditional architectures good enough? Guido Schmutz BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH