Name InfoSphere Streams

Transcription

1 Name InfoSphere Streams Title: Jaskiran Bhatia Country Manager Information Management

2 Rules! Mining! Custom Analytics!...In micro seconds

3 . & these opportunities are everywhere Transportation Intelligent traffic Management Stock market Impact of weather on securities prices Analyze market data at ultra-low latencies Health & Life Sciences Neonatal ICU monitoring Epidemic early warning system Remote healthcare monitoring

4 . & these opportunities are everywhere Law Enforcement, Defense & Cyber Security Real-time multimodal surveillance Situational awareness Cyber security detection e-science Space weather prediction Detection of transient events Synchrotron atomic research Telephony CDR processing Social analysis Churn prediction Geomapping

5 Traffic Control System in City of Stockholm Data sources GPS from 1000 s taxis Loop Sensors Speed of traffic Flow density of traffic (cars per second) CCTV video inside tunnels Real Time Weather data Output Travel time forecasts Via SMS Now, In 30 minutes, 1 hour, 2 hours etc Integrate with existing system

6 Traffic Management for Sustainability and Efficiency Multimodal Data Streams GPS Cell-phones (location tracking) Public Transport (bus, docking) Pollution measurements Weather Conditions (including road conditions) Optical traffic flow detectors Travel time data based on plate recognition Induction loop detector data Accidents in network as they are being recorded Road closures (road work, etc) Still pictures from road cameras GPS Data Streams Real Time Transformation Logic Real Time Geo Mapping Real Time Speed & Heading Estimation Real Time Aggregates & Statistics Real Time Traffic Monitoring & Information (Multimodal) Travel Planner Interactive visualization Storage adapters Only 4 x86 Blade servers to process 250,000 GPS probes per second, maps of 630,000 line segments Web Server Google Earth Data Warehouse Offline statistical analysis

7 Real Time Geo Mapping & Speed Estimation GPS probe Matching map artifact Estimated path Estimated speed & heading

8 Recommendations Based on Hurricane Forecast Capture market data (high Compute portfolio Correlate volume) market indicators combined risk and trade VWAP to (low latency) determine buy/sell Capture weather recommendations sensor data, analyses hurricane Estimate predicted path impact on portfolios Make recommendations Dynamically DHTML updated Result rendering and notify risk assessment for assets in projected path System S platform Web Zero platform Real-time projections of hurricane path

9 World s fastest options trading prototype Identify and execute trades Process over 5M events per second with average latency of 150 microseconds Expand to incorporate content feeds, news text, audio, video, to establish greater context for better decisions CIO TD Bank "TD Bank Financial Group worked with IBM Research to develop a first-of-akind architecture capable of consuming, analyzing and acting on real-time market data while maintaining sub-millisecond response times even under extreme data loads

10 Equities Trading Starter Application Modular design Components are plug-replaceable extend these or substitute your own Demonstrates how trading strategies may be swapped out at runtime, without stopping the rest of the application TradingStrategy module looks for opportunities that have specific quality values and trends OpportunityFinder module looks for opportunities and computes quality metrics SimpleVWAPCalculator module computes a running volume-weighted average price metric

11 Predictive Analytics using InfoSphere Streams in a neo natal ICU helps detect life threatening conditions upto 24hrs earlier Real Time analytics and correlations on physiological data streams Blood pressure, Temperature, EKG, Blood oxygen saturation etc., Early detection of the onset of potentially life threatening conditions Upto 24 hours earlier than current medical practices Early intervention leads to lower patient morbidity and better long term outcomes Technology also enables physicians to verify new clinical hypotheses

12 Law Enforcement and Security Federal Government Streams of information including video surveillance, wire taps, communications, call records, etc. Millions of streams per second with low density of critical data Identify patterns and relationships among vast information sources "The US Government has been working with IBM Research since 2003 on a radical new approach to data analysis that enables high speed, scalable and complex analytics of heterogeneous data streams in motion. The project has been so successful that US Government will deploy additional installations to enable other agencies to achieve greater success in various future projects" - US Government

13 SPSS Modeler to Build Model, Streams to Detect Quickly Characterization of Motive Build rulesets ( profiles ) of various cause categories Utilizing crime scene information such as... REAL-TIME ANALYTIC PROCESSING Crime reports when entered

14 Government and Law Enforcement: e911 Support Data from 911 calls, satellite feeds, imagery from city traffic cameras Streams defines the geo spatial location of the call by running powerful analytics in real time using satellite communication link and draws in city camera feeds from around the area Real time support for 911 dispatcher and field personnel Data stored for future auditing and evidence requirements

15 Sharpe Engineering and US Navy Phase 1 SBIR Research Navy Sensors + + = InfoSphere Streams Advanced Analytics Identify/build sample problem Preliminary sizing Est. development, deployment and operation costs Possible use cases Maritime commerce & Anti piracy Unmanned surface vehicles Disaster relief Cyber security Command & Control Sharpe Engineering

16 From Data Analytics To Smarter Cyber Security IDS Firewall Security Events Event Normalization Historical data summaries &evidence Temporal Anomalies, Event / Destination Correlations, Partial periodicity IPS/ADS Sensors Data Repository: Log aggregation / normalization Temporal Analysis True Positives/ False Positives Human annotations Manual Tuning DNS Live Data Security Analytics, e.g., Botnet Analytics InfoSphere Warehouse Channel Profiles ID & NAC Entity Analytics (GNR,...) Botnet Models App/DB Logs ADS Models Self-tuning Feedback Unsupervised Supervised Statistical Models

17 Forecasting Space Weather at LOFAR Outrigger in Scandinavia (LOIS) Solar Flares Triaxial Antenna InfoSphere Streams Radio signal input and data preparation Signal detection and noise filtering + + = Strength and 3D directional analysis Space Weather prediction regarding impact on satellites and electric grids Swedish Institute of Space Physics

18 Telco Moving to Agile, Real-time Processes & Analytics Large Indian Wireless Telco million customers. >10% annual growth. Expanding operations abroad and growing to provide real-time services and 3G capabilities to customers / Business Value Corporate Visibility Marketing Campaign & Service Analytics Enterprise Data Warehouse, Data Marts & Reporting Operational Efficiency Cross-sell/Up-Sell, Reduced Activation Time Customer Analytics and Business Process Management Data Infrastructure Optimization Single, real-time data feed for Fraud, BI & Revenue Assurance systems Real-time CDR processing Information Management Evolution Reduce Complexity Manage Risk Reduce Cost Reduce data latency from 6-12 hours to seconds Implemented fault tolerant, and flexible solution Consolidated existing integration systems by 50% Improvement in data processing throughput Streamlined development & maintenance of data services

19 Streams provides tight integration with existing Information / Analytics Infrastructure for Call Detail Record Processing Real Time User Analytics SOA Web Service Cognos Mashups Applications Spreadsheets Fin Planning InfoSphere Warehouse Data Marts DB2 Pre-processed Data Info Server Analytic Models CDRs InfoSphere Streams ERP,CRM and Other Data Sources

20 Real Time Charging system telco in Japan 1. The Pain Point Real time charging can increase revenue/profit But: 20 million users Dramatic call and IP traffic growth SMS grows at 30% annually Even hourly summaries were hard to do 2. The Solution InfoSphere Streams with soliddb processes: 55K CDRs per second on 1 octicore node 10 million in 200 seconds 160K CDRs per second on 3 octicore nodes 10 million in 60 seconds Nearly linear growth Architectural pattern to prevent data loss Demonstrated high software productivity 3. The Happy Ending Telco is extending the pilot; in production in 1H 2011 Platform to create new real time billing system