Complex Event Processing: Power your middleware with StreamInsight Mahesh Patel (Microsoft) Amit Bansal (PeoplewareIndia.com)
Agenda The Value of Timely Analytics The challenges / Scenarios Introduction to Complex Event Processing Introducing Microsoft StreamInsight Brief introduction to BizTalk Server Understand the key scenarios that StreamInsight can unlock for BizTalk Map out the architectural, integration requirements for BizTalk + StreamInsight.
The value of timely analytics
Current Products for analytics
Current Products for analytics
Introduction to CEP Complex Event Processing (CEP) is an architecture style based on the principles of Event Driven Architecture (EDA). Continuously processing a high-volume stream of events from various event sources Events are logically grouped in sequences (typically known as streams) based on a time defined criteria such as an interval. Event streams are processed through a series of queries
Capabilities of a CEP Application Event Driven Continuous Queries Time Sensitive Low Latency High Volume
The value of timely analytics
Data Stream Data Stream Scenarios Manufacturing: Sensor on plant floor React through device controllers Aggregated data 10,000 events/sec Web Analytics: Click-stream data Online customer behavior Page layout 100,000 events /sec Financial Services: Stock & news feeds Algorithmic trading Patterns over time Super-low latency 100,000 events /sec Power, Utilities: Energy consumption Outages Smart grids 100,000 events/sec Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds Visual trend-line and KPI monitoring Batch & product management Automated anomaly detection Real-time customer segmentation Algorithmic trading Proactive condition-based maintenance Asset Specs & Parameters Event Processing Engine Stream Data Store & Archive Lookup Threshold queries Event correlation from multiple sources Pattern queries
Introducing StreamInsight
CEP using StreamInsight DEMO
Messages and Events Large Messages Low Volume High Volume Small Events LOB B 2B Web Services RFID Electrical Grids Stock Feeds CRM SWIFT Health Care Operational Data Traditional messaging applications: High signal every message is business relevant Rich data schemas sets of related information Workflow centric and transactional Emerging data sources and trends: Machine born data is growing at a very rapid rate Not all of this data is business relevant (low signal/noise ratio) How do we: Identify and extract business relevant events from streaming data? Take action on these business insights (line of business, human workflow, etc)
LOB Basic BizTalk Architecture Message Host Host Host Message Receive Port Orchestration Send Port Adapter (HTTP) Send Port Adapter (File) Receive Pipeline Send Pipeline Component Component Maps Message Agent Map Message Agent Message Agent Management DB MessageBox Tracking DB
Scenarios Good BizTalk + StreamInsight scenarios: Streaming data/event sources high noise to signal ratio Need to analyze and extract business relevant events Integrate and transact with line of business applications.
Scenarios Manufacturing (Oil and Gas) Monitor plant floor processes Identify business relevant events Transact with LOB apps Utilities Automated metering handle flood of data from utility meters Identify business relevant events Transact with LOB apps The data feeds change, the question changes the core scenario doesn t!
OSIsoft PI & StreamInsight OSIsoft PI - Industry leading process data management and historian. Integrating and leveraging StreamInsight OSIsoft shipping StreamInsight adapter for PI. Stream data in and out of PI Extract powerful business insights from PI data Integrate with LOB applications using BizTalk Server
OPC & StreamInsight (with OPC Industry standard for process automation and data exchange MatrikonOPC Industry leader in OPC interfaces MatrikonOPC) Developed StreamInsight adapters for their OPC implementation Extract powerful business insights from OPC data Integrate with LOB applications using BizTalk Server
Automated Metering Contoso Utilities is rolling out smart electrical meters to their customers. Meters report power usage (wattage, voltage) at a point in time. Data is consolidated into a back end application used for billing and forecasting. The volume of data has forced Contoso to only read values from meters every hour They want to: Monitor the data more closely, compressing the point stream to an interval stream before loading into the back end application. i.e. if the data does not change by more than 2% over a 10 minute window, keep the average value. Capture spikes and threshold events, route to a process control application for further inspection.
Automated Metering Collect high volume streaming data from smart meters Data consists of timestamp, source ID and payload (power in Watts) Sample operations on the streaming data: Consolidate stream of points events into 5 minute chunks ; record average, count and standard deviation. Detect anomalies (threshold 10%), collect last 5 minutes worth of data for that meter and all meters fed from that substation Need to link meter ID with reference data (location, substation) Results must be integrated into line of business application Timestamp 3/5/2010 14:23 Power 1020 Meter ID 112322 Data consolidation (point -> interval) Anomaly Detection Reference data (location, hierarchy) Batch to Message Batch to Message Line of Business Application Smart Meters
Automated Metering Smart Meters Timestamp 3/5/2010 14:23 Power 1020 Meter ID 112322 1 2 Data consolidation (point -> interval) Anomaly Detection 3 Reference data (location, hierarchy) Batch to Message 4 Batch to Message 5 Line of Business Application 1 Meter Input Adapter (StreamInsight) Consolidation Query Threshold Detection Query StreamInsight 2 3 Message Builder Output Adapter Query Results (XML) 4 WCF MSMQ Adapter Receive Port 5 Orchestration BizTalk Server Send Port LOB Adapter Query Result Format (XSD)
WCF Adapters WCF Adapters Architecture Message Integration XmlPayload Output Adapter Queries Input Adapters 1 3 Batched Query Output BizTalk Host Orchestration StreamInsight XSD Schema 1 StreamInsight extracts insights from real time data feeds, feeds aggregate information to BizTalk. 2 A strongly typed XML schema is inferred from the StreamInsight query 2 Business Rules Engine Message Box Business Activity Monitoring 3 Batched and correlated query results are delivered to a BizTalk receive port.
Deployment Message Integration MSMQ BizTalk Hosts StreamInsight Instance SQL Servers for BizTalk StreamInsight is an in-memory event processing engine. Do not deploy StreamInsight instance on the same hardware as your BizTalk host instances. Leverage StreamInsight-local MSMQ for message transfer.
Summary BizTalk Server and StreamInsight together can unlock powerful scenarios: High volume data processing Message integration Leverage StreamInsight to: Consolidation and analyze incoming data streams convert event sets into messages Deliver to BizTalk receive port Leverage BizTalk Server to: Integrate these insights into line of Business applications Other potential alignment points: Business Activity Monitoring (BAM). Tap into BAM event events stream for rich analytics and event detections Business Rules Engine (BRE). Leverage BRE to define event pattern rules
2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.