From Information to Foresight:

Size: px

Start display at page:

Download "From Information to Foresight:"

Neal Manning
6 years ago
Views:

1 Laura Haas, IBM Fellow IBM Research - Almaden From Information to Foresight: Getting Beyond the Bits IBM Corporation

Data sources are growing, and new sources are emerging 2

Climate 220 Terabytes of Web data 9 Petabytes of

6 Billon Mobile Phones World Wide New Sources of

day Facebook processes 10 terabytes of data every day 74

processing DB Traditional Sources of Information UPS

2 Data sources are growing, and new sources are emerging 2 Billion Internet users by 2011 World Data Centre for Climate 220 Terabytes of Web data 9 Petabytes of additional data 4.6 Billon Mobile Phones World Wide New Sources of Information Twitter processes 7 terabytes of data every day Facebook processes 10 terabytes of data every day 74 Terabytes UK Land Registry - Largest known transaction processing DB Traditional Sources of Information UPS supports largest known peak database workload of 1.1 Billion SQL statements per hour 2 2 Sources: Guardian, IDC, IBM 2011 IBM Corporation

3 The Challenge: Bring together large volumes of diverse data to find new insights Multi-channel customer sentiment and experience analysis Detect life-threatening conditions at hospitals in time to intervene Predict weather patterns to plan optimal wind turbine usage, and optimize capital expenditure on asset placement Make risk decisions based on real-time transactional data Identify criminals and threats from disparate video, audio, and data feeds IBM Corporation

Online Transaction Processing System Co gn os +

grappling with increasingly complex data and

Feedback/Action Str u ClickSteam, CRM Claim

Location Tracking (GPS), iphone, Vehicle Use

high volume information (evolving, highly

Financial Planning Scorecards Feeds: Census

data 4 i-str Sem uct Semi-s truct Billions of

reputation analysis, behavior analytics,

Predictive Analytics 100 s TBs, PetaBytes Deep

4 Online Transaction Processing System Co gn os + + Data analytics in the enterprise means grappling with increasingly complex data and sophisticated analytics faster than ever Search Feedback/Action Str u ClickSteam, CRM Claim data (text, picture, video) Call data records Location Tracking (GPS), iphone, Vehicle Use Data, $ Trans tracking (Across borders & IP providers), Embedded Analytics ctu red Dashboards Semi-Un-struct Continuous arrival of high volume information (evolving, highly variant) (struct-/semi--/un-structured Mash ups Financial Planning Scorecards Feeds: Census Bureau Data Market Data, Weather Data Sensors data 4 i-str Sem uct Semi-s truct Billions of mobile devices Social Media for web buzz, reputation analysis, behavior analytics, intent, Auto/Cross Correlation Analytics, Predictive Analytics 100 s TBs, PetaBytes Deep & Wide Analytics Fine grained individual product and customer at a time and place IBM Corporation

5 Preparing data for analysis itself requires analytics Q: How exposed am I to my borrowers? SEC Midas Flow Search UI FDIC Crawl Extract Resolve Map & Fuse Temporal Analysis Financial Companies & Key People Reports Loan borrower, lender Event Company bankruptcies, merger/acquisitions employment, director, officer insider, 5% owner, 10% owner job changes Person subsidiaries, insider, 5%, 10% owner, banking subsidiaries issuer Security holdings, transactions holdings, transactions IBM Corporation

SEC Midas Flow Search UI FDIC Crawl Extract Resolve Map & Fuse

Report Loan Agreement Insider Transaction Proxy Statement

6 Preparing data for analysis itself requires analytics Q: How exposed am I to my borrowers? SEC Midas Flow Search UI FDIC Crawl Extract Resolve Map & Fuse Temporal Analysis Financial Companies & Key People Reports Annual Report Loan Agreement Insider Transaction Proxy Statement Unstructured, Data spread over many documents! Loan Exposure Counterparty Relationships Integrated Entities and Relationships IBM Corporation

7 Programmable analytics are needed for extraction and resolution Metadata : Tracy Guida Tracy Tampa just a Nor-Cal gal trying to fall in just a Nor-Cal gal trying to fall in love with Florida love with Florida Messages : Screen name Screen name Location: Tampa, Fl Location: Tampa, Fl Name: Tracy Guida Name: Tracy Guida IsParent: true IsParent: true Gender: female Gender: female Relationships: Relationships: <@thelorenski, acquaintance > <@thelorenski, acquaintance > <@SparklingLala, acquaintance> <@SparklingLala, acquaintance> <@smileycookie, network> <@smileycookie, network> Interests: sports: (Yoga, running, Steelers), food:( Interests: sports: (Yoga, running, Steelers), food:( Burger21) Burger21) so blessed to have a son that loves so blessed to have a son that loves school! #backtoschool school! Thanks for being a loyal Thanks for being a loyal fan, Tracy! :) #Burger21 Tracy! :) great! I thanks again for coming in great! I thanks again for coming in last will be there Nov for night, looking forward to Yoga will be there Nov for night, looking forward to Yoga a conference, will keep in Sunday a conference, will keep in Sunday I am coming to SJ in Jan, be warned :) coming to SJ in Jan, be warned :) Sore after my run this Sore after my run this morning absolutely! I am the biggest Steelers absolutely! I am the biggest Steelers fan living in Tampa. Going to share these cookies with fellow living in Tampa. Going to share these cookies with fellow Steelers fans. :) Steelers fans. :) IBM Corporation

Relationships: <@thelorenski, acquaintance > <@thelorenski, acquaintance > <@SparklingLala, acquaintance> <@SparklingLala, acquaintance> <@smileycookie, network> <@smileycookie, network> Interests:

8 Programmable analytics are needed for extraction and resolution Profile Name: Champagne Girl Same local ID across posts: " " No valid name, no location Reference to twitter screen name Screen name Screen name Location: Tampa, Fl Location: Tampa, Fl Name: Tracy Guida Name: Tracy Guida IsParent: true IsParent: true Gender: female Gender: female Relationships: Relationships: <@thelorenski, acquaintance > <@thelorenski, acquaintance > <@SparklingLala, acquaintance> <@SparklingLala, acquaintance> <@smileycookie, network> <@smileycookie, network> Interests: sports: (Yoga, running, Steelers), food:( Interests: sports: (Yoga, running, Steelers), food:( Burger21) Burger21) Global Profile: Screen name Screen name Location: Tampa, Fl Location: Tampa, Fl Name: Tracy Guida Name: Tracy Guida IsParent: true IsParent: true Gender: female Gender: female Relationship: Relationship: <@thelorenski, acquaintance > <@thelorenski, acquaintance > <@SparklingLala, acquaintance> <@SparklingLala, acquaintance> <@smileycookie, network> <@smileycookie, network> <Siena Brennick, friend> <Siena Brennick, friend> Interests: sports: (Yoga, running, Steelers), food:( Interests: sports: (Yoga, running, Steelers), food:( Burger21, Pinkberry, wine, Burger21, Pinkberry, wine, Bubba s Buffalo Chicken Dip) Bubba s Buffalo Chicken Dip) Blogs: Source: myohercitybythebay.com, Text =. Blogs: Source: myohercitybythebay.com, Text = IBM Corporation

9 Such analytics require a rich platform with design and runtime tools Solutions Telco Finance CPG Health Govt. Job Scheduling High Level Query (Jaql) Entity Extraction (System T) Adaptive Map Reduce Machine Learning (System ML) Enterprise Scale File System (GPFS-SNC) Analytics Libraries indexing Ingest/Connect (Web crawl, Warehouse, ETL...) Workload Optimization Export (BI, Visualization) Innovation above, underneath, in and around Hadoop IBM Corporation

Maximum insight requires combining deep and reactive analytics Exa Deep Analytics Data Scale Peta Data Scale Tera Giga Hypotheses Integration Deep History Integration Predictions Feedback Directly

10 Maximum insight requires combining deep and reactive analytics Exa Deep Analytics Data Scale Peta Data Scale Tera Giga Hypotheses Integration Deep History Integration Predictions Feedback Directly integrating Reactive and Deep Analytics enables feedback-driven insight optimization Reality Mega Traditional Data Warehouse and Business Intelligence Observations Fast Actions Integration Reactive Analytics Kilo yr mo wk day hr min sec ms µs Occasional Frequent Real-time Decision Frequency IBM Corporation

11 There are many kinds of analytics, often domain-specific Stochastic Optimization Optimization Predictive modeling How can we achieve the best outcome including the effects of variability? How can we achieve the best outcome? What will happen next if? Prescriptive Forecasting What if these trends continue? Predictive Simulation What could happen.? Alerts What actions are needed? Actionable Value Query/drill down Ad hoc reporting Standard Reporting What exactly is the problem? How many, how often, where? What happened? Descriptive Degree of Complexity IBM Corporation

Today s challenges require integration of models and analytics Education Agriculture Advertising Transportation Health care Payers practices Recreational SPLASH: Smarter Planet Platform for Analysis

12 Today s challenges require integration of models and analytics Education Agriculture Advertising Transportation Health care Payers practices Recreational SPLASH: Smarter Planet Platform for Analysis and Simulation of Health Demographics Health care Providers Economics Disease models Pharma companies facilities Health care Clients Supply Chain Government policymakers Immigration Treatment standards Patient perception and satisfaction Cultural and Social influences Mashup BMI estimation Access to Physical activities Store Location Model Transportation model Eating & Exercise Model City Incentive model Queuing model Optimization Time series Agent based Formula Identify previously unclear connections between parts of complex healthcare systems Demographic, behavioral, GIS data sources for initialization of models IBM Corporation

13 Combining analytics is easier with the help of a compiler Inquiry Compiler Ease of use Shorter Time to Deployment Reusable Software Components IBM Corporation

14 Visualization and collaboration are key to understanding data IBM Corporation

15 The Institute for Massive Data, Analytics and Modeling Taking Analytics to the Limit Enable solution of real-world problems Drive new breakthroughs across stack Leverage unique IBM (Research) capabilities World-class platform for data-intensive research Research Centers Commercial Partners IBM Research Institute for Massive Data, Analytics & Modeling Government Agencies Academia People: Global Research Expertise Software: IBM Products & Research Assets Data: At rest, in motion, structured and unstructured Infrastructure: Massive Compute, Storage & Network Healthcare Energy Disaster Business Risk IBM Corporation

16 Summary Better collection of data affords many opportunities To do better science To improve our world To make more money To benefit, we need to do more than collect data! Data preparation is a challenging problem Tooling and runtimes are both important Model integration is a key emerging challenge Supporting collaboration and understanding both data and analysis is essential One of the key challenges for gaining insight is integrating and enabling people The Institute for Massive Data, Analytics and Modeling offers exciting possibilities For computer scientists: drive new innovation in information management & mining For domain researchers: leverage data for new discoveries in that field For the world: new techniques to solve big problems Through bringing together data, technology and people with diverse skills IBM Corporation

Harnessing the Power of Big Data to Transform Your Business Anjul Bhambhri VP, Big Data, Information Management, IBM

May, 2012 Harnessing the Power of Big Data to Transform Your Business Anjul Bhambhri VP, Big Data, Information Management, IBM 12+ TBs of tweet data every day 30 billion RFID tags today (1.3B in 2005)