Copyright 2015, Oracle and/or its affiliates. All rights reserved.

Size: px

Start display at page:

Laureen Potter
6 years ago
Views:

2 Finding new business potential with Big Data Analytics Carsten Frisch Oracle Business Analytics DOAG 2015 Business Solutions Conference Darmstadt, 10. Juni 2015 Copyright 2015, Oracle and/or its its affiliates. All All rights reserved.

3 Referent» Carsten Frisch» Senior Sales Consultant» Business Analytics Big Data Discovery Lead - DE/CH Cluster» Kontakt +49 (0) » carsten.frisch@oracle.com Copyright 2015, Oracle and/or its affiliates. All rights reserved.

It is intended It is not for a commitment information purposes to deliver only, any material, and may not code, be or incorporated functionality, into and any should contract.

4 Safe Harbor Statement Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information The following purposes is intended only, to and outline may our not general be incorporated product direction. into any contract. It is intended It is not for a commitment information purposes to deliver only, any material, and may not code, be or incorporated functionality, into and any should contract. not be It is relied not a upon in commitment making purchasing to deliver decisions. any material, The development, code, or functionality, release, and and timing should of not any be features relied upon or functionality in making purchasing described decisions. for Oracle s The products development, remains release, at the and sole timing discretion of any of Oracle. features or functionality described for Oracle s products remains at the sole discretion of Oracle. Copyright 2015, Oracle and/or its affiliates. All rights reserved. 4

Monetizing New Insights Business Cases for Big Data and the Discovery

Enabling Rich Customer Experience Across

have become more Email Mail demanding and

The customer experience expectations for

reset by the experiences Online Mobile being

8 Enabling Rich Customer Experience Across Channels Is A Key Focus For Banks Customers have become more Mail demanding and their loyalties Sales are diffused with low-switching Branch 360 degree view of customer Phone costs. The customer experience expectations for banking services (across channels) are being reset by the experiences Online Mobile being provided by retailers and ATM online providers elsewhere Source: Redefining Customer Experience, Infosys Whitepaper; PWC Report 2012 Copyright 2015, Oracle and/or its affiliates. All rights reserved. 8

Banks Need To Move Towards Personalization And Targeted Marketing To Enhance Customer Experience Top 3 Emerging Changes in Customer Behavior That Impact Banking (% of respondents) Using Direct and

9 Banks Need To Move Towards Personalization And Targeted Marketing To Enhance Customer Experience Top 3 Emerging Changes in Customer Behavior That Impact Banking (% of respondents) Using Direct and Self-Service Channels Seeking Better, More Personal Advice Price Sensitivity, Discount Seeking 49% 44% 63% Customer Demand. More personalized services, offers and enhanced customer experience. More relevant services and transparent access to information across all channels consistently. Increase simplicity, self-control, mobility of banking services Customers are making web / mobile as their primary channel of interaction with their banks. These channels are already heavily personalized and there is a rising demand for more personalized services and offers from customers Source: Enhancing The Banking Customer Value Proposition Through Technology-led Innovation, Accenture Copyright 2015, Oracle and/or its affiliates. All rights reserved. 9

Market Challenges Are Compelling Banks To Focus On Customer Insight And Real-Time Offers INDUSTRY CHALLENGES ENHANCE CUSTOMER EXPERIENCE REAL-TIME OFFERS KEY BIG DATA CAPABILITIES Develop deep client

spending habits, and segmentation to gain view on customer risk and enhance risk management capabilities Generate real-time, context sensitive, targeted offers based on analytical insights on

Financial Services Industry Solutions Overview; Oracle Insight; PWC Report Provide more visibility into performance in order to facilitate timely and cost effective management of operations Discover

10 Market Challenges Are Compelling Banks To Focus On Customer Insight And Real-Time Offers INDUSTRY CHALLENGES ENHANCE CUSTOMER EXPERIENCE REAL-TIME OFFERS KEY BIG DATA CAPABILITIES Develop deep client relationships by offering superior service Analyze internal customer logs and social media activity to generate indications of customer dissatisfaction allowing time to act Analyze behavior profiles, spending habits, and segmentation to gain view on customer risk and enhance risk management capabilities Generate real-time, context sensitive, targeted offers based on analytical insights on spending patterns Rapid time to market and improved customer value Leverage insights from social media during various stages of product and service development OPTIMIZE OPERATIONS Source: Oracle Financial Services Industry Solutions Overview; Oracle Insight; PWC Report Provide more visibility into performance in order to facilitate timely and cost effective management of operations Discover opportunities to achieve greater efficiency across global operations Understand and forecast performance and drive strategies that improve operations and financial results Copyright 2015, Oracle and/or its affiliates. All rights reserved. 10

11 Leveraging Big Data for Competitive Advantage in FS Customer Insight Data Monetisation Optimise Operations Customer Insight Social Media Sentiment & Engagement Big Data Augmentation Real Time Offers Context Sensitive Offers / Ads Location Based Offers / Ads Compliance Processes Personalised Services New Product Launch New Revenue Streams Information as a Service Fraud Detection Risk Management Fast Data Quality of Models Financial risk Security risk Copyright 2015, Oracle and/or its affiliates. All rights reserved.

Characteristics of Digital Business Leaders They Reframe

angles They Sprint They work at pace - researching, testing and

That Failure Can Be Good and are not afraid of new ideas They

data and data from external sources to establish patterns and

13 Characteristics of Digital Business Leaders They Reframe Challenges Looking at them from new perspectives and multiple angles They Sprint They work at pace - researching, testing and evaluating current ideas while generating new ones They Appreciate That Failure Can Be Good and are not afraid of new ideas They Convert Data Into Value They invest heavily in analyzing their own data and data from external sources to establish patterns and un-noticed opportunities Copyright 2015, Oracle and/or its affiliates. All rights reserved. 13

14 Identify (business) question Verify earlier findings Design of a solution model Gather all necessary data Analyse the data Present & implement results Data-driven Decisions Become clear about all aspects of the decision to be taken or the problem to be solved. Try to identify alternatives to your perception Non-Analysts & Executives: should take a closer look on steps 1 and 6 of the analysis process if they plan to make use of statistical analysis. Find out who has investigated such or a similar problem in the past and the approach that has been taken Formulate a detailled hypothesis how specific variables might influence the result of the chosen model Gather all available information about the variables of your hypothesis. The relevance of a dataset might address your business question directly or needs to be derived Apply a statistical model and evaluate the correctness of the approach. Repeat this procedure until the right method has been identified. Data Science + Knowledge Discovery Frame the results obtained in a comprehensible story. This kind of presentation intends to motivate decision makers and relevant stake-holders to take action Adopted from Thomas H. Davenport, Harvard Business Manager 2013 Copyright 2015, Oracle and/or its affiliates. All rights reserved. 14

15 Vertical and Horizontal Data Science Skills Data Vertical Warehouse Deep technical skills Eigenvalues, Lasso-related regressions Experts in Bayesian networks, R Support Vector Machine Hadoop, NoSQL, Data Modeling, DW Machine Learning & Statistics Storytelling experts Horizontal Cross-discipline knowledge Visualization skills Programming experience Domain expertise Aware of pitfalls & rules of thumb Look for the individual Unicorn or build a Data Science Team? The Specialist The Unicorn Copyright 2015, Oracle and/or its affiliates. All rights reserved. 15

Enabling Data-driven Innovations in Organizations Business Analysts: Day-to-Day performance of a business unit Information Consumer: Reporting on individual transactions Automated Process: Decisions

Knowledge Discovery Dynamic Dashboards and Reports Volume and Fixed Reporting BICC ACC Knowledge Driven Business Process Insight Analytical Competence Center (ACC)» Separate group reporting to CxO.

16 Enabling Data-driven Innovations in Organizations Business Analysts: Day-to-Day performance of a business unit Information Consumer: Reporting on individual transactions Automated Process: Decisions effecting execution of an indiv. transactions Executive: Decisions effecting strategy and direction Data Scientists: Information analysis to meet strategic goals Perf. Mgmt. Knowledge Discovery Dynamic Dashboards and Reports Volume and Fixed Reporting BICC ACC Knowledge Driven Business Process Insight Analytical Competence Center (ACC)» Separate group reporting to CxO. not part of a Business Intelligence Competence Center (BICC)» Mission: broadening the adoption of Analytics across the organization» Skilled resource pool of Data Scientists, Statisticians and Business Experts» Data-driven approach (not development-driven) with privileged access to enterprise data sources» Group will be assigned to projects for a limited time Copyright 2015, Oracle and/or its affiliates. All rights reserved. 16

18 Information Management Conceptual View Actionable Events Actionable Insights Actionable Information BICC Structured Enterprise Data Data Streams Event Engine Data Reservoir Data Factory Enterprise Information Store Business Intelligence Other Data Execution Innovation Line of governance ACC Events & Data Discovery Lab Discovery Output Source: Oracle White Paper Information Management and Big Data A Reference Architecture Copyright 2015, Oracle and/or its affiliates. All rights reserved. 18

Discovery Lab: Design Pattern ACC» Iterative development approach data oriented NOT development oriented» Small group of highly skilled individuals (aka Data Scientists or a team organized as an

19 Discovery Lab: Design Pattern ACC» Iterative development approach data oriented NOT development oriented» Small group of highly skilled individuals (aka Data Scientists or a team organized as an Analytical Competence Center, ACC) with privileged access to enterprise data sources» Specific focus on identifying commercial value for exploitation» Wide range of tools and techniques applied» Typically separate infrastructure but could also be unified Reservoir if resource managed effectively» Data provisioned through Data Factory or own ETL processes Copyright 2015, Oracle and/or its affiliates. All rights reserved. 20

Discovery Lab: Activity Cycles Copyright 2015,

Virtualisation & Information Services Discovery Lab: Data Provisioning Analysis Processing & Delivery Data Factory flow General BI

or new channels 1 Pre-Built Intelligence Assets Intelligence Analysis Tools Dashboards & Reports Scorecards Charts & Graphs Ad Hoc

Sandbox Project 3 Query & Search Tools Raw Data Sandbox Project 2 Statistics Tools Sandbox Project 1 Data store Analytical Processing

quickly develop new reporting through mashups from any available internal and external sources and may used advanced analytical tools

21 Virtualisation & Information Services Discovery Lab: Data Provisioning Analysis Processing & Delivery Data Factory flow General BI flow The majority of BI development activity will be from existing sources performed by the BICC developing new reports to existing or new channels 1 Pre-Built Intelligence Assets Intelligence Analysis Tools Dashboards & Reports Scorecards Charts & Graphs Ad Hoc Query & Analysis Tools OLAP Tools Forecasting & Simulation Tools BICC Reporting Tools 2 Discovery Lab & Development Environment Sandbox Project 3 Query & Search Tools Raw Data Sandbox Project 2 Statistics Tools Sandbox Project 1 Data store Analytical Processing Data Science (Primary Toolset) Data Modelling Tools Programming & Scripting Data & Text Mining Tools Faceted Query Tools ACC ACC may quickly develop new reporting through mashups from any available internal and external sources and may used advanced analytical tools for innovative analysis Data Quality & Profiling Graphical rendering tools Copyright 2015, Oracle and/or its affiliates. All rights reserved. 22

Polystructured Data Structured Data Unified: Big Data Management and

Exalytics Exadata Oracle BI Foundation Suite (ROLAP/MOLAP, Mobile, )

Enterprise) Oracle Database Oracle SQL Queries Oracle Big Data SQL

discoveries in Big Data Discovery» Publish results to the Hadoop

Oracle R for Hadoop Experiment, Prototype & Collaborate BDA Data

R for Hadoop SQL join Tables in Hadoop Productize, Secure, Govern»

SQL» No data movement required» Seamlessly extends existing DWH and BI

22 Polystructured Data Structured Data Unified: Big Data Management and Analytics Experiment, Prototype, Collaborate Productize, Secure & Govern Exalytics Exadata Oracle BI Foundation Suite (ROLAP/MOLAP, Mobile, ) In-Memory Appliance Oracle Advanced Analytics (Data Mining, Oracle R Enterprise) Oracle Database Oracle SQL Queries Oracle Big Data SQL Tables in DB» Quickly find, explore, transform, analyze and share discoveries in Big Data Discovery» Publish results to the Hadoop Distributed File System (HDFS)» Use to build predictive models with Oracle R for Hadoop Experiment, Prototype & Collaborate BDA Data Warehouse Oracle Big Data Discovery Hadoop (HDFS) Data Reservoir Oracle R for Hadoop SQL join Tables in Hadoop Productize, Secure, Govern» Connect published HDFS files to secure Oracle DB using Oracle Big Data SQL» No data movement required» Seamlessly extends existing DWH and BI investments with non-traditional data in Hadoop Copyright 2015, Oracle and/or its affiliates. All rights reserved. 23

designed for Hadoop» Emerging solutions lack broad capabilities 80% effort typically spent on evaluating and

23 Need To Get Analytic Value Fast Data Uncertainty» Not familiar and overwhelming» Potential value not obvious» Requires significant manipulation Tool Complexity» Early Hadoop tools only for experts» Existing BI tools not designed for Hadoop» Emerging solutions lack broad capabilities 80% effort typically spent on evaluating and preparing data Overly dependent on scarce and highly skilled resources Copyright 2015, Oracle and/or its affiliates. All rights reserved. 24

26 Oracle Big Data Discovery: Components Hadoop Cluster (Oracle Big Data Appliance or Commodity Hardware with Cloudera CDH 5.) BDD node name node data node data node data node data node Hadoop 2.x Metadata (HCatalog) Workload Mgmt (YARN) Filesystem (HDFS) Studio Oracle Big Data Discovery Workloads Web UI: Find, Explore, Transform, Discover, Share In-Memory Discovery Indexes DGraph: Search, Guided Navigation, Analytics Data Processing, Workflow & Monitoring Profiling: catalog entry creation, data type & language detection, schema configuration Sampling: dgraph (index) file creation Transforms: >100 functions Enrichments: location (geo), text (cleanup, sentiment, entity, key-phrase, whitelist tagging) Self-Service Provisioning & Data Transfer Personal Data: Upload CSV and XLS to HDFS Other Hadoop Workloads MapReduce Spark Hive Pig Oracle Big Data SQL (Oracle Big Data Appliance only) Copyright 2015, Oracle and/or its affiliates. All rights reserved. 27

Oracle Big Data Discovery: Preparation of Data Sources Have to be

Table with a standard Regex SerDe ( Serializer-Deserializer ) to

into regular table columns Hive Table definition for fixed-width

map nested file structures of a JSON file into regular table

27 Oracle Big Data Discovery: Preparation of Data Sources Have to be created as Hive Tables and registered in the Hive Metastore Hive Table with a standard Regex SerDe ( Serializer-Deserializer ) to map more complex file structures by using Regular Expressions into regular table columns Hive Table definition for fixed-width or delimited files Hive Table using a custom developed SerDe to map nested file structures of a JSON file into regular table columns Copyright 2015, Oracle and/or its affiliates. All rights reserved. 29

Big Data Discovery Upload of XLS und CSV files and

Experience) Upload of various file formats, table

Command Line Interface is similar to the MySQL command

28 Oracle Big Data Discovery: Preparation of Data Sources There are multiple ways to get new Data Sets loaded Big Data Discovery Upload of XLS und CSV files and automatic Hive Table creation HUE (Hadoop User Experience) Upload of various file formats, table creation wizzards, web-based Hive Query Client Hive Command Line Interface is similar to the MySQL command line Copyright 2015, Oracle and/or its affiliates. All rights reserved. 30

29 Oracle Big Data Discovery: Preparation of Data Sources or by using your favorite Data Integration / ETL Tool File (FS/HDFS) IKM File To Hive (Load Data) IKM Hive Transform IKM Hive Control Append Hive LKM HBase to Hive Hive IKM File-Hive To Oracle (OLH, OSCH) Oracle DB IKM SQL to Hive- HBase-File (SQOOP) HBase IKM Hive to HBase Hive IKM File-Hive to SQL (SQOOP) Any RDBMS Hive HBase Any RDBMS Oracle Data Integrator with Advanced Big Data Option (Supporting HDFS, Hive, HBase, Scoop, Pig, Spark) Copyright 2015, Oracle and/or its affiliates. All rights reserved. 31

rows Profiling and Enrichment Process access_logs 1 m rows BDD access_logs 1 m

30 Oracle Big Data Discovery: Data Ingestion Data Processing Workflow including Profiling and Enrichment access_logs 100m rows Hive / HCatalog access_logs 1 m rows Profiling and Enrichment Process access_logs 1 m rows BDD access_logs 1 m rows 1M of 100M Copyright 2015, Oracle and/or its affiliates. All rights reserved. 32

Catalog» Access a rich, interactive catalog of all data in Hadoop» Familiar search and guided navigation for ease of use» See data set summaries, user

32 Catalog» Access a rich, interactive catalog of all data in Hadoop» Familiar search and guided navigation for ease of use» See data set summaries, user annotation and recommendations» Provision personal and enterprise data to Hadoop via selfservice Copyright 2015, Oracle and/or its affiliates. All rights reserved. 36

33 Explore» Visualize all attributes by type» Sort attributes by information potential» Assess attribute statistics, data quality and outliers» Use scratch pad to uncover correlations between attributes Copyright 2015, Oracle and/or its affiliates. All rights reserved. 37

Transform» Intuitive, user driven data wrangling» Extensive library of powerful data transformations and enrichments» Preview results, undo, commit

34 Transform» Intuitive, user driven data wrangling» Extensive library of powerful data transformations and enrichments» Preview results, undo, commit and replay transforms» Test on sample data then apply to full data set in Hadoop Copyright 2015, Oracle and/or its affiliates. All rights reserved. 38

36 Transform but flexible (based on Groovy Programming Language / Library) Preferred Method for IT / Data Engineer / Data Scientist Copyright 2015, Oracle and/or its affiliates. All rights reserved. 40

37 Discover» Join and blend data for deeper perspectives» Easy usage - compose project pages via drag and drop» Use powerful search and guided navigation to ask questions» See new patterns in rich, interactive data visualizations Copyright 2015, Oracle and/or its affiliates. All rights reserved. 41

38 Share» Share projects, bookmarks and snapshots with others» Build galleries and tell big data stories» Collaborate and iterate as a team» Publish blended data to HDFS for leverage in other tools Copyright 2015, Oracle and/or its affiliates. All rights reserved. 42

42 % of Positive Responders Analytics: More Data Variety available Better Results Example: Marketing Campaigns Getting lift on responders Data Mining-based prediction results with Response Modelling including hundreds of input variables like:» Demographic data» Purchase POS transactional data» Polystructured data, text & comments» Spatial location data» Long term vs. recent historical behaviour» Web visits» Sensor data» Population Size (% of Total Cases) Naïve Guess or Random Model with 20 variables Model with 75 variables Model with 250 variables 100 Copyright 2015, Oracle and/or its affiliates. All rights reserved. 46

Oracle Advanced Analytics Native SQL Data Mining/Analytic

(ORE)» Allows distributed processing of huge data volumes»

Security and SQL access» R Studio = GUI for Data Analysts

kernel» Direct access via PL/SQL API and SQL operators» Oracle

43 Oracle Advanced Analytics Native SQL Data Mining/Analytic Functions + High-performance R Integration Oracle R Enterprise (ORE)» Allows distributed processing of huge data volumes» Benefits from DB features, e.g. Security and SQL access» R Studio = GUI for Data Analysts Oracle Data Mining (ODM)» Implemented in the Oracle Database kernel» Direct access via PL/SQL API and SQL operators» Oracle Data Miner GUI embedded in SQL Developer Copyright 2015, Oracle and/or its affiliates. All rights reserved. 47

Oracle Big Data Discovery The Visual Face of Big Data

Oracle Big Data Discovery The Visual Face of Big Data Today's Big Data challenge is not how to store it, but how to make sense of it. Oracle Big Data Discovery is a fundamentally new approach to making