Future of Data Hortonworks Data Platform and Hortonworks Data Flow. Eric Thorsen, VP Industry Solutions

Size: px

Start display at page:

Download "Future of Data Hortonworks Data Platform and Hortonworks Data Flow. Eric Thorsen, VP Industry Solutions"

Hannah Hopkins
5 years ago
Views:

1 Future of Hortonworks Platform and Hortonworks Flow Eric Thorsen, VP Industry Solutions

Explosion of Consumer Behavior Millennials now outnumber Baby Boomers as dominant transactional generation and will constitute 50% of workforce in next few years, 75% of workforce by 2030 2.

2 Explosion of Consumer Behavior Millennials now outnumber Baby Boomers as dominant transactional generation and will constitute 50% of workforce in next few years, 75% of workforce by Billion Connected People on Social networks by 2020, 75 Billion Connected Devices by 2020 Big Trends The number of U.S. firms using big data has jumped 58 percentage points to 63% penetration 70% of firms now say that big data is of critical importance to their firms, from only 21% in One of the fastest techadoption rates ever. The title of chief data officer the C-Suite manager of big data a title that until recently didn t even exist, is now found in 54% of companies surveyed. Exploding with unprecedented data types Sensors, ibeacons, Weighted Shelves, Smart Hangers, Smart Bins, Smart Racks Social Media, Tweets, Mentions, Likes, Blogs Clickstream, Web logs, video feeds Server activity 2 Hortonworks Inc All Rights Reserved Big Executive Survey 2016 NewVantage Partners 80% of the world s data has been created in the last two years. Ginni Rometty, IBM CEO January 2014

3 DATA More Volume and More Types E X A B Y T E S USER CLICK STREAM USER GENERATED CONTENT MOBILE WEB SOCIAL NETWORK SENTIMENT B I G D A T A P E T A B Y T E S T E R A B Y T E S OFFER HISTORY OFFER DETAILS WEB LOGS A/B TESTING DYNAMIC PRICING SEGMENTATION C R M AFFILIATE NETWORKS W E B SEARCH MARKETING EXTERNAL DEMOGRAPHICS BUSINESS DATA FEEDS HD VIDEO SPEECH TO TEXT PURCHASE DETAIL ERP CUSTOMER TOUCHES BEHAVIORAL TARGETING PRODUCT/ SERVICE LOGS G I G A B Y T E S PURCHASE RECORD PAYMENT RECORD SUPPORT CONTACTS DYNAMIC FUNNELS SMS/MMS I N C R E A S I N G D A T A V A R I E T Y A N D C O M P L E X I T Y 3 Hortonworks Inc All Rights Reserved

4 1,000,000,000,000,000,000,000 1 Traditional systems under pressure Challenges Constrains data to app Can t manage new data Costly to Scale Business Value 2 New Zettabytes Much of the new data exists in-flight between systems and devices as part of the Internet of Anything Zettabytes New Clickstream Geolocation Web Internet of Things Docs, s Server logs *Multiples of Bytes Kilobyte Megabyte Gigabyte Terabyte Petabyte Exabyte Zettabyte Yottabyte ERP CRM SCM Traditional 4 Hortonworks Inc All Rights Reserved Zettabytes

5 Hortonworks Delivers Connected Platforms Modern Applications ACTIONABLE INTELLIGENCE PERISHABLE INSIGHTS HISTORICAL INSIGHTS INTERNET OF ANYTHING DATA IN MOTION DATA AT REST Hortonworks Flow Hortonworks Platform 5 Hortonworks Inc All Rights Reserved

6 Hortonworks Connected Platforms and Solutions Hortonworks Solutions Enterprise Warehouse Optimization Cyber Security and Threat Management Internet of Things and Streaming Analytics Hortonworks Connection Hortonworks Connection Subscription Support SmartSense Premier Support Educational Services Professional Services Community Connection Center Hortonworks Suite HDP HDF Cloud Hortonworks Cloud AWS HDInsight 6 Hortonworks Inc All Rights Reserved

7 Holistic Customer Interaction Model Customer Proposal Components Configure, Manage and Upgrade Support/ Break Fix HDP and HDF Subscription Components Included Operational Services Applications Professional Services and Partner SI s 7 Hortonworks Inc All Rights Reserved

8 Hortonworks Influences the Apache Community We Employ the Committers --one third of all committers to the Apache Hadoop project, and a majority in Apache NiFi and other important projects Our Committers Innovate and improve Connected Platforms We Influence the Hadoop Roadmap by communicating important requirements to the community through our leaders A P A C H E H A D O O P C O M M I T T E R S 8 Page 8 Hortonworks Hortonworks Inc Inc All Rights Reserved All Rights Reserved

9 Payment Tracking Due Diligence Sentiment Social Mapping Customer Support Optimize Inventories Next Product Recs Cyber Security Call Machine Product Design M & A Ad Placement Basket Segments Proactive Repair Disaster Mitigation Investment Planning Factory Yields Defect Detection Cross- Sell Supply Chain Customer Retention Vendor Scorecards Inventory Predictions Risk ModelingAd Placement OPEX Reduction Device Ingest as a Service Historical Records Fraud Prevention Mainframe Offloads Rapid Reporting Digital Protection Public Capture 9 Page 9 Hortonworks Inc All Rights Reserved

Payment Tracking Due Diligence Sentiment Social Mapping Customer Support Optimize Inventories Next Product Recs Cyber Security Call Machine Product Design M & A Ad Placement Basket Segments Proactive

10 Payment Tracking Due Diligence Sentiment Social Mapping Customer Support Optimize Inventories Next Product Recs Cyber Security Call Machine Product Design M & A Ad Placement Basket Segments Proactive Repair Disaster Mitigation Investment Planning Factory Yields Defect Detection Cross- Sell Supply Chain Customer Retention Vendor Scorecards Inventory Predictions Risk ModelingAd Placement OPEX Reduction Device Ingest as a Service Historical Records Fraud Prevention Mainframe Offloads Rapid Reporting Digital Protection Public Capture 10 Page 10 Hortonworks Inc All Rights Reserved

11 The Journey to a Golden Batch

M A N U FA C T U R I N G E X P L O R E O P T I M I Z E T R A N S F O R M Product Monitoring Process Monitoring Customer I N N OVAT E D ATA D I S C OV E RY Quality S I N G L E V I E W Product Process

12 M A N U FA C T U R I N G E X P L O R E O P T I M I Z E T R A N S F O R M Product Monitoring Process Monitoring Customer I N N OVAT E D ATA D I S C OV E RY Quality S I N G L E V I E W Product Process Predictive Maintenance P R E D I C T I V E A N A LY T I C S Real-time Operations Time Series MFG Quality Batch Genealogy Connected Car Equipment Production Line Proactive Repair Yield Optimization Quality Optimization Yield Warranty Logistics Factory Supply Chain Business Inventory Predictions Supply Chain Optimization Energy Management R E N OVAT E Demand Sensing OPEX Reduction A C T I V E A R C H I V E Device Ingest E T L O N B O A R D D ATA E N R I C H M E N T as a Service Fraud Prevention Mainframe Offloads Rapid Reporting Digital Protection Public Capture 12 Hortonworks Inc All Rights Reserved

13 Merck s Journey Improving Life Sciences Manufacturing Yields Presents a Complex Discovery Challenge Vaccine manufacturing requires precise control of complex fermentation processes Two batches of a vaccine, produced using an identical manufacturing process, can exhibit significant yield variances Batches that fail quality standards can cost $1 million each Merck analyzed one vaccine: 10 years of manufacturing data stored across 16 systems 13 Hortonworks Inc All Rights Reserved

14 Merck s Journey Innovate Renovate Sensor Storage Scientific Search D A T A D I S C O V E R Y Vaccine Yield Optimization A C T I V E A R C H I V E D ATA D I S C O V E R Y Epidemiology D A T A D I S C O V E R Y The Golden Batch The Journey to the Golden Batch Combined 10 years data amounted to 1 billion records 5.5 million batch comparisons 1 st year yield boost of 40K more doses $10M profit impact McKinsey: 50% yield improvement 14 Hortonworks Inc All Rights Reserved

15 The Journey to Safe Roads

I N S U R A N C E E X P L O R E O P T I M I Z E T R A N S F O R M Risk Assessment Due Diligence Sentiment Solvency I N N OVAT E D ATA D I S C OV E RY Social Mapping S I N G L E V I E W Customer

16 I N S U R A N C E E X P L O R E O P T I M I Z E T R A N S F O R M Risk Assessment Due Diligence Sentiment Solvency I N N OVAT E D ATA D I S C OV E RY Social Mapping S I N G L E V I E W Customer Support Fraud Investigation Next Best Action P R E D I C T I V E A N A LY T I C S Cyber Security Call Sensor Product Design M & A Ad Placement Claim Market Segments Loss Control Cat Models Investment Planning Telematics Loss Control Cross- Sell Risk Customer Retention Channel Scorecards Claim Severity Fraud Mitigation Risk Modeling Risk Appetite R E N OVAT E Historical Records OPEX Reduction A C T I V E A R C H I V E Device Ingest E T L O N B O A R D D ATA E N R I C H M E N T as a Service Fraud Prevention Mainframe Offloads Rapid Reporting Digital Protection Public Capture 16 Hortonworks Inc All Rights Reserved

17 Progressive s Journey Progressive Wanted to Ingest IoT to Predict Risk for its Usage-based Insurance Product Progressive Snapshot offers usage-based insurance through an in-car sensor that transmits IoT driving data Sensors collect up to six months of data from drivers and the data is archived for years, per regulatory requirements Progressive s existing systems were not scaling efficiently It took 5 7 days to transform only 25% of available UBI data 17 Hortonworks Inc All Rights Reserved

Progressive s Journey D A T A D I S C O V E R Y Web Log Innovate Renovate Online Ad Placement D A T A D I S C O V E R Y ETL O N B O A R D Sensor Ingest Usage-Based Insurance (UBI)

Safety Snapshot plug-in devices capture driving detail, 100% stored in HDP, ingested in 2-3 days More than 12 billion miles driven stored Through a web app, customers can review their

18 Progressive s Journey D A T A D I S C O V E R Y Web Log Innovate Renovate Online Ad Placement D A T A D I S C O V E R Y ETL O N B O A R D Sensor Ingest Usage-Based Insurance (UBI) PREDICTIVE ANALYTICS Claims Notes Mining A C T I V E A R C H I V E D A T A D I S C O V E R Y Individual Driving Histories Safe Roads Rewarding Safer Drivers and Improving Traffic Safety Snapshot plug-in devices capture driving detail, 100% stored in HDP, ingested in 2-3 days More than 12 billion miles driven stored Through a web app, customers can review their own driving detail and improve their safety Snapshot and usage-based insurance drove $2.6 billion in 2014 Progressive premiums, growing since then 18 Hortonworks Inc All Rights Reserved

19 The Journey to Discover the Genetic Links to Cancer

20 Arizona State University s Journey A Genome Represents 20 Billion Rows of and Researchers Couldn t Explore Enough Genetic to Understand How Genes Affect Cancer Cancer is both complicated (the interplay between multiple biological process) and also complex (affected by biological, genetic, environmental and social factors) Since each genome represents so much data, legacy platforms couldn't amass enough genomic data to explore cancer patterns across a broad genetic spectrum This created a lamp-posting phenomenon, forcing a focus around incremental research clustered around genes known to influence cancer ASU turned to HDP to store and process huge amounts of genomic data, to make that data broadly available to researchers and to do it all at a scalable cost 20 Hortonworks Inc All Rights Reserved

21 Arizona State University s Journey HDP s Storage and Compute Efficiencies Allow Individual Researchers to Do the Work Previously Done By Entire Teams ASU s Next Generation Cyber Capability (NGCC) project combines HDP with high-performance computing, for genomic analysis in Apache Spark The NGCC architecture follows President Obama s National Cancer Moonshot guidelines, with a federated framework that encourages data sharing One query against a table with 20 billion rows would time out before it could return results. In HDP it returned results in 1-2 minutes Now with HDP we have both the availability of data and the technical capability to analyze it. We are able to explore spaces where we simply couldn t go before. It just wasn t possible before having this technology. This has sped our time to insight infinitely in some cases. Some questions were not possible before, and now they return results in a day. -- Dr. Kenneth Buetow, Director of Computational Sciences and Informatics 21 Hortonworks Inc All Rights Reserved

22 The Journey to Better Health

H E A LT H C A R E E X P L O R E O P T I M I Z E T R A N S F O R M Claims Optimization HCAHPS Scores Sentiment Proactive Outreach I N N OVAT E D ATA D I S C OV E RY Physician Notes Quality Benchmarks

23 H E A LT H C A R E E X P L O R E O P T I M I Z E T R A N S F O R M Claims Optimization HCAHPS Scores Sentiment Proactive Outreach I N N OVAT E D ATA D I S C OV E RY Physician Notes Quality Benchmarks Device Monitoring Care-path Best Practices Cohort Selection Social Sentiment R & D Bill Shock OR Optimization STARS Ratings Patient Throughput S I N G L E V I E W 360 Patient View Seasonal Staffing Patient Experience Supply Chain Net Promoter Score Patient Outreach Customer Churn Medication Safety Inventory Preventative Medicine P R E D I C T I V E A N A LY T I C S Staffing Predictions Proactive Maintenance Remote Monitoring Genomics Census Drug Diversion R E N OVAT E Lab Notes Archive OPEX Reduction A C T I V E A R C H I V E Device Ingest E T L O N B O A R D D ATA E N R I C H M E N T as a Service Fraud Prevention Imaging Archive Legacy System Mainframe Offloads Historical Patient Records Rapid Reporting Digital Protection Real-time Decision Support Improved Drug Yields 23 Hortonworks Inc All Rights Reserved

24 Mercy s Journey Mercy Medical System Sought a Lake for a Single View of its Patients One Patient, One Record Existing platform impeded goal of enriching Epic data for 1 million patients over 35 Hospitals and 500 clinics Moving Epic EMR data to Clarity EDW took 24 hours and was never going to enable real-time analytics. Now that takes 3-5 minutes with HDP. Improved billing processes resulted in $1M additional annual revenue from newly documented secondary diagnoses and care 24 Hortonworks Inc All Rights Reserved

25 Mercy s Journey OPEX Efficiency D A T A D I S C O V E R Y A C T I V E A R C H I V E Lab Notes Innovate Renovate Billing S I N G L E V I E W A C T I V E A R C H I V E Vital Signs D A T A D I S C O V E R Y Epic EMR Replication A C T I V E A R C H I V E Single Patient Record S I N G L E V I E W Privacy base Preventive Care D A T A E N R I C H M E N T PREDICTIVE ANALYTICS Epic Enrichment P R E D I C T I V E A N A L Y T I C S Device Ingest Medical Decision Support ETL O N B O A R D Better Health Better Health Through Searches of free-text lab notes, speed researcher insight from never to seconds Ingest of ICU vital signs increased by 900X, letting clinicians respond more quickly Mercy is building real-time tools to support surgical decisions and preventive care 25 Hortonworks Inc All Rights Reserved

27 Governance & Integration Security Operations Hadoop for Retail ANALYSIS Single view of consumer Targeted promotions Recommendation engines Basket analysis Price optimization Inventory optimization Loyalty management Path to purchase DATA REPOSITORIES EDW Script SQL NoSQL Stream Search In-Mem Others ERP YARN : Operating System CRM RDBMS 1 HDFS (Hadoop Distributed File System) N TRADITIONAL SOURCES EMERGING & NON-TRADITIONAL SOURCES CRM STORES PRODUCT CATALOG STAFFING PLANS SOCIAL MEDIA IN-STORE WIFI LOGS SENSOR RFID 27 Hortonworks Inc All Rights Reserved ERP POS TRANSACTIONS INVENTORY WEB TRANSACTIONS BEACONS CLICKSTREAM SERVER LOGS

29 Start Your Journey with Hortonworks in 5 Easy Steps 1. Schedule a use case workshop with the local Hortonworks team 2. Complete the Big Scorecard hortonworks.com/get-started/big-datascorecard 3. Download Hortonworks Sandbox hortonworks.com/sandbox 4. Subscribe for Support hortonworks.com/services/jumpstart/ 5. Join Hortonworks Community Connection hortonworks.com/community/ 6. Follow the Hortonworks blog hortonworks.com/blog/ 29 Hortonworks Inc All Rights Reserved