Data Driven Culture Enabled by Data Management

Size: px

Start display at page:

Download "Data Driven Culture Enabled by Data Management"

Emil Nash
5 years ago
Views:

1 IBM Analytics Academy Data Driven Culture Enabled by Data Management John J Thomas Distinguished Engineer & Director IBM Analytics Jo Ramos Distinguished Engineer & Director IBM Analytics 1

Data is what Fuels Digital Transformation 75% Only 15% Value of large enterprises will have digital transformation at the center of corporate strategy within

2 Data is what Fuels Digital Transformation 75% Only 15% Value of large enterprises will have digital transformation at the center of corporate strategy within two years YET of business leaders are getting what they need from their data to drive their transformations Sources: Forbes, McKinsey, MIT Sloan. Most are Here 2

3 Most Are Accelerating Their Move Up the Data Maturity Curve Operations BI and Data Warehousing Self-Service Analytics New Business Models Value AI Most are here COST REDUCTION MODERNIZATION INSIGHT-DRIVEN TRANSFORMATION Maturity 3

4 AI Promises to Unlock the Value of Data The most disruptive new capability over the next decade 85% believe AI will deliver competitive advantage $39B in new AI investments in x increase in demand for ML/AI skills 78% will climb the ladder to AI, or have already started 4

amount of algorithmic sophistication will overcome a lack of data.

5 Success Starts with Making Your Data Ready for AI Data collection and preparation are most time-consuming activities in deploying AI Need good data lifecycle governance practices No amount of algorithmic sophistication will overcome a lack of data... bad data is simply paralyzing Enterprise class Analytics capabilities are needed 81% don t understand the data needed for AI 5

There is no AI without an IA INFUSE - Automate and scale across your processes TRUST - Achieve trust and transparency in outcomes ANALYZE - Scale insights with ML everywhere ORGANIZE - Create a

6 There is no AI without an IA INFUSE - Automate and scale across your processes TRUST - Achieve trust and transparency in outcomes ANALYZE - Scale insights with ML everywhere ORGANIZE - Create a trusted analytics foundation COLLECT - Make data simple and accessible Multicloud Data Architecture Data of every type, regardless of where it lives MODERNIZE your data estate for an AI and multicloud world 6

7 Make Your Data Ready for AI App Developers Data Engineers Data Stewards Data Scientists Business Users & Analysts MAKE DATA SIMPLE AND ACCESSIBLE Collect every type of data no matter where it lives BUILD A TRUSTED ANALYTICS FOUNDATION Organize your data into a trusted source of truth SCALE INSIGHTS ON DEMAND Empower all your teams to analyze data in smarter ways Data Deploy a Cloud-native data architecture AI Reimagine your workflows with AI 7

An Information Architecture for AI is Critical Collect Hybrid Data Management Organize Unified Governance & Integration Analyze Data Science & Business Analytics Collect all types of data, structured

System Big SQL & Hortonworks Hadoop Satisfy all matters of finding, cataloging and masking data Integrates fluid data sets Delivers built-in compliance Leverages advanced machine learning

8 An Information Architecture for AI is Critical Collect Hybrid Data Management Organize Unified Governance & Integration Analyze Data Science & Business Analytics Collect all types of data, structured and unstructured Includes all open sources of data Leverages a single platform with a common application layer Write once and deploy anywhere Db2 & Db2 Warehouse Db2 Event Store Integrated Analytics System Big SQL & Hortonworks Hadoop Satisfy all matters of finding, cataloging and masking data Integrates fluid data sets Delivers built-in compliance Leverages advanced machine learning capabilities Delivers descriptive, prescriptive and predictive insights across all types of data Empowers all your teams and their unique use cases Enables advanced analytics and data science methods User & application independence across on premise, private cloud, and public cloud 8

9 IBM Cloud Private for Data Seamlessly integrated platform supporting end-to-end Data & Analytics workflows Collect, Connect, and Access Data Govern, Search, and Find Data Understand and Prepare Data for Analysis Build Descriptive, Predictive, and Prescriptive Models & Reports Model Management and Deployment Create Analytics Applications Connect and discover content from multiple data sources across your organization. Provision databases and federate data access. Auto classify. Ingest metadata, auto assign terms and rules. Grant user access levels and enforce business policies. Index for search, visualize consumers and producers of assets with lineage, metrics, and quality profiles. Find data and analytics assets in the Catalog. Understand, cleanse and prepare your data to create data preparation pipelines visually. Use popular open source frameworks to prepare structured and unstructured data. Scale data integration and transformation on performant engines. Create Machine Learning, Deep Learning, Optimization, and other advanced mathematical models. Design your models programmatically or visually with popular open source tooling and IBM frameworks. Visualize data. Create dashboards and business reports. Manage your models across dev, test, staging, and prod. Deploy your models and scale automatically for online, batch or streaming use cases with SLAs. Monitor model performance and automatically trigger retraining and redeployment as rolling upgrades. Incorporate trusted and governed models into applications, dashboards, and operational systems.

10 IBM Cloud Private for Data - Architecture Open, Extensible API Platform Personalized, Collaborative Team Platform App Developers Data Engineers Business Partners Data Stewards Data Scientists Business Users Instant, Pre-assembled Provisioning Admin & Ops Dashboards Cloud-native Data Micro Services Collect Data Organize Data Analyze Data Databases on-demand Data warehousing Fast data ingest Federated query Data integration Data curation Governance, privacy, policies Data asset lifecycle management Data visualization & exploration Machine learning & deep learning Model management & deployment Dashboards & business reporting Enterprise Data Catalog Enterprise Data Catalogs IBM Cloud Private Kubernetes Platforms Elastic Runtimes Product Ready Operations Built-in Continuous Delivery Secure to the Core 10

11 Data Management IBM and Open Ecosystem Invisible Governance, Db2 Portfolio, MongoDB, Postgres, Data Virtualization, etc. Multi-Cloud Unified Console Optimized for fast and flexible data service lifecycle management Ease of portability across private & public clouds One experience across all data services Coherent, efficient, and scalable data services experience Ease of provisioning, monitoring, and more Ecosystem & Virtualization Db2 Warehouse, OLTP, and Event Store Partnerships with MongoDB, Postgres, etc. Data Virtualization Data Governance & Self-Service Analytics Data automatically integrated with governance capabilities for autodiscovery, catalog, and search subject to policies & rules enabling self-service data discovery

12 The path to governed data & analytics Governance omnipresent, invisible, and infused throughout Data Sources Systems of Record Systems of Engagement 3 rd Party Data Social Media Documents News Auto Data Discovery Extract, Transform, Cleansing, Standardize Define Data Governance Objectives Profile: Data Classify & Term Assg Auto Data Quality Enforce (Policies & Rules) Data Curation & Stewardship Measure & Monitor Intelligent Metadata Catalog (infused with ML) Metadata Catalog Data Quality Management Regulatory & Compliance Reporting Shopping for Information Machine Learning & Self-Services Analytics Weather Other External The path to governed data BI Reporting Dashboards

13 Data Science A Day in the Life Development & Prototyping Production Business Executive & Data Science Leader Sets business goals Prioritizes use cases Manages project plans Measures results Understand Use Case Find Data Data Steward & Data Engineer Collect & Organize: Data & Analytics Asset Catalog Lineage Governance of Data & Models Audits & Policies App Developer Infuses ML in apps & business processes Rolling Upgrades Deploy Model, Monitor Health, & SLAs Data Scientist Analyze: Explores, shapes data Features engineering and extraction Use the right hypothesis to build the algorithm, trains models Validates results correlating to the use case Explore & Understand Data Collect & Prepare Data Production Secure & governed Monitor & instrument High throughput Load balance and scale Reliable deployment - outage free upgrades Train Model Extract Features Data Engineer Organize & Integrate: Builds & enables data pipelines Collaborates with product teams to integrate results

14 Data Science Platform Best-of-breed tooling from open ecosystem Authoring tools Machine learning, deep learning, optimization Customize environments, packages & images Authoring Tools Coding and visual modeling options Machine Learning Runtimes Deep Learning Runtimes Cloud private infrastructure Container-based resource management Scale with distributed and GPU support IBM Cloud Private Infrastructure Model Lifecycle Management Dev -> Test -> Staging -> Prod Versioning, release, SLAs, rolling upgrades Model Lifecycle Management IBM Cloud / 2018 IBM Corporation

Gather and Manage Data Search for Data Introduce Databases and Data Lakes Create and store data in a relational database (e.g., Db2 Warehouse) Add relational databases (Oracle, Db2.

15 Gather and Manage Data Search for Data Introduce Databases and Data Lakes Create and store data in a relational database (e.g., Db2 Warehouse) Add relational databases (Oracle, Db2..) and Hadoop systems to the catalog Enable discovery & classification and metadata search indexes Initiate discovery and classifications Work with Analytics Projects (as a sandbox) Introduce csv files or remote data assets Collaborate inside Projects Refine and explore data via visualizations Invite others to collaborate

16 Understand Data Work within Analytics Project Sandboxes Refinery tool Scripts & notebooks (Python Pandas, R etc.) Explore & Refine Understand distribution of data Sample and profile Shape and wrangle Build Dashboards Find Data (search for data) Explores & Understands Data, distributions Understand data distributions & profile Refined Data Sets Sample data Programmatically build out visualizations & data wrangling

Ready for Analytics Build repeatable & schedulable transformation

17 Data Preparation Create data transformation pipelines Prep & Transform Data Collect Builds data lakes and warehouses Gets Data Ready for Analytics Build repeatable & schedulable transformation jobs Work with different types of data sources A rich set of transformation operators

Govern Data Data Steward Organizes Data & Analytics Asset Catalog Lineage Governance of Data & Models Audits & Policies Enables Model Explainability Discover &

18 Govern Data Data Steward Organizes Data & Analytics Asset Catalog Lineage Governance of Data & Models Audits & Policies Enables Model Explainability Discover & add to catalog Classify assign terms Quality Analysis & Policies Trusted Data For Analytics A Catalog of all data assets Understand relationships between entities

19 Build ML Models Explores & Understands Data Distributions - ML Feature Engineering - Visualizations - Notebooks - Train Models - Dashboards & Apps Models, Scoring Services & Apps Features, Lineage recorded in Governance Catalog Publish to the Catalog Jupyter notebook Environment Python 2.7/3.5 with Anaconda Scala, R R Studio Environment with > 300 packages, R Markdown, & R Shiny Data Scientist Zeppelin notebook with Python 2.7 with Anaconda Analyzes Experiments, trains models Evaluates model accuracy Publish apps & models

20 Model Management & Deployment Operationalize Analytics & ML Run scoring service at scale, load-balance Monitor effectiveness Version & upgrade models in production App Developer Infuses ML in apps & business processes Trusted Predictions Monitor & Manage Automate with Jobs trigger on-demand or by a schedule. such as for Model Evaluations, Batch scoring or even continuous (re-) training Production Secure & governed Monitor & instrument High throughput Load balance and scale Reliable deployment - outage free upgrades Deploy & monitor accuracy Models, Scoring Services & Apps Auto-retrain & upgrade

21 Extensibility an ecosystem for partners & developers Drill-downs & deep integration to H2o Flow Example H2o notebooks managed as first class assets. H2o Environments - scale up with GPU affinity/requests H2o Environments (add-on) - self-provisioning by each data scientist

Scaling AI with Trust and Transparency in the Enterprise Your AI At scale With Trust and Transparency IBM TOOLS IBM AI RUNTIME Infuse and Manage AI at scale Watson Studio Watson Machine Learning

22 Scaling AI with Trust and Transparency in the Enterprise Your AI At scale With Trust and Transparency IBM TOOLS IBM AI RUNTIME Infuse and Manage AI at scale Watson Studio Watson Machine Learning Project OpenScale 3 RD PARTY IDE & FRAMEWORKS 3 RD PARTY RUNTIMES Watson Services Natural Language Classifier Visual Recognition Fairness Explainable AI Continuous Evolution Orchestration Model Health Accuracy Infuse Into Apps Build Run Manage Data Scientist Software Engineer AI Operations In app use Business User

23 IBM GCDO CEDP Hybrid Cloud Showcase Building an enterprise data platform: Challenges Technology Organizational considerations Business processes Organizational How to overcome siloed organization roadblocks Business Processes How to resolve inconsistent master data management across processes while also addressing regulatory requirements IBM s Chief Data Office developed & implemented: Data The AI Enterprise An AI enterprise platform Business and governance transformation to truly enable an AI enterprise. Technology How to build a scalable, secure, high capability data platform Data How to efficiently capture metadata for data platform assets as ingested and derived data exponentially grows

24 IBM had to address the same demands as the rest of the market Relentlessly reduce unit IT costs by optimizing workloads across a multi-cloud architecture. Usage patterns for each workload, heuristics from each cloud, and embedded ML will deliver an optimized recommendation to reduce unit costs. Easily and securely store, process and find value in a continuous volume of structured and unstructured data. ML-infused data governance Continually improve and personalize experiences by infusing AI and Machine Learning. Rapidly design, deploy and maintain applications in a multi-cloud environment. Using IBM solutions we delivered enterprise grade security, performance, and scalability AI and ML infused in the engine and in persona-based end user experiences Adoption of open source technologies like Docker containers and Kubernetes orchestration enables dev ops methodology and agile approaches 24 24

25 IBM GCDO CEDP Hybrid Cloud Showcase The Business Imperatives: One common Enterprise data backbone that is at the core of every business process transformation A single version of the truth: one place for Enterprise relevant data, enabling consistency at enterprise scale A single Platform that provides data, computing, and analytics/ai : this is the key to scaling AI and business process innovation across the Enterprise A Platform built on IBM Cloud Private (ICP) that allows seamless movement of workloads between Private and Public clouds The Platform Offering ~100 data assets totaling 1PB leveraging 50+ production services Data Ingestion: billions of records ingested/updated daily (batch and streaming) Automated Metadata Generation ML / DL ranked recommendations Security/Governance leveraging IBM enterprise tools and GDPR compliant Entity/Record Linkage: hundreds of million entity pairs linked in seconds Large scale Deep Learning networks trained on TBs of data in 6-8 hours Contextual ML Recommender Engines trained in seconds Hundreds of data engineers/data scientists/developers rapidly implementing ~20 use case solutions, with 6 already in production enabling 6K IBM professionals: Individual Client Insights (ICI): holistic insights and recommendations for 20M+ client contacts Client360: holistic insights and recommendations for 1M+ IBM clients IBM Services Cognitive Contracts: client delivery "early warning system"/scope expansion recommendation engine Risk Insights: combining geo/weather, supplier and events data to proactively mitigate risk events on IBM Supply Chain Blue Seller: Smart Chatbot addressing questions from sellers & other personas Expert Configurator Assistant: streamlined client configurator development IBM Corporation 25

IBM GCDO CEDP Hybrid Cloud Showcase Architectural Design Full Stack Power8/9 for DL CEDP compute infrastructure combines commodity compute nodes with High Performance IBM Power9/8 servers 10 s of

26 IBM GCDO CEDP Hybrid Cloud Showcase Architectural Design Full Stack Power8/9 for DL CEDP compute infrastructure combines commodity compute nodes with High Performance IBM Power9/8 servers 10 s of GPUs being used for training large Deep Neural Networks on TBs of data in parallel 6-8 hours training cycles. All Flash Data Lake CEDP is built on high performance scalable storage infrastructure. It combines PBs of IBM ESS Flash and HDDs on 100Gb ETH - RDMA network. Delivering 92.5GB/s reads & 51.5GB/s writes on 2 ESS nodes, thus enabling both high throughput(for GPUs) and low latency (for DB workloads). Hybrid Data Management Seamless integration/movement of workloads across a IBM Cloud Private for Data & Watson Studio. Built on IBM Cloud, IBM Flash Storage and PowerAI Unified Governance & Data Integration *Services available in the IBM Cloud Private for Data Services available / supported in IBM Cloud Private for Data offering available in 2Q18; Public Cloud showcases Fall Data Science & Business Analytics True Hybrid Cloud IBM Cloud Private for Data is built on best of breed IBM and open source technologies. It provides an integrated environment for data ingestion, preparation, and AI modeling. DSX to build and deploy large scale ML and DL models for predictive modeling and recommender engines in hours Public Cloud Backup Highly sensitive data Initial Deep Learning models The CEDP Private Cloud consists of combination of VMs and Containers on IBM Cloud Private (ICP). ICP is the bridge between Private and Public cloud enabling seamless portability of workloads to IBM Cloud (Public). IBM Corporation 26

27 Stakeholder Pain Points Motivations for an integrated platform IT expense model Financial Sponsors CFO VP Finance Project risk-adjusted return Cloud-native architecture Security, governance, and compliance CxO Chief Information Officer Chief Data Officer VP VP Analytics/IT VP Line of Business Day-to-day project management Integrating outputs from working teams Keeping systems operational Analytics Teams: Data Scientists, Data Engineers, App Developers Responsible for capturing the value of AI Data Stewards Responsible for data governance System Administrators Responsible for systems & infrastructure Cross-functional collaboration & skills Data ingest Data access and federation Data quality, lineage, policies Reproducible data pipelines Machine learning & optimization Analytic asset sharing and cataloging Model management and SLAs Model trust, quality, and retraining Enforcing business policies Compliance with regulation Tagging data assets Annotating data assets Validating data quality Validating data lineage Tracking and protect data usage Provisioning capabilities Configuring, upgrading, and automating Allocating and managing resources Monitoring system health Ensuring uptime and performance Maintaining system security Maintaining multiple product stacks Ensuring parity between dev, test, and prod environments

28 How - AI Transformation Build a Strategy AI is the new Digital Disruption 2017 McKinsey Articulate Use Case Break Down Data Silos Identify AI Technologies, Partner or Acquire tools to fill technology gaps. Integrate AI in your workflow, between business units Adopt an open and collaborative culture. 28

29 3 Key Strategies to drive AI for Business.. 1 Skills - Build a balanced team Programmatic and visual data scientist Enable business domain experts 2 Organization Operationalize Use Cases Experimentation to deployment to increase business value Cross-train and increase team velocity Create Center of Excellence and empower lines of business 3 Tools - Connect data, algorithms and applications Access, prep and analyze data seamlessly All tools in one environment Open ecosystem 29

30 Discussion and Q&A 30

31 sy y B a y l d M g b Bh C i d a e l d a l cd d ha m a i l Mi a aa a CC g C i d d am a i C i d l d am a i a w o e o Cg C i a l M l d d p n l Mi o l Mi l d d a l cd d ha m t a w d l im g i l Mid ha m i e a ld d id C i B d a i Mi l d ha m a i Mi a i i a Mi l d d a l cd d ha m r B sl am Mi e r B i t t B i s l c am g id l g i Mid ha m im B g B l C a g B d M u B r B I i ham.. / / / / / / M B w a i ham... l I 2018 IBM Corporation 3

Paul Chang Senior Consultant, Data Scientist, IBM Cloud tw.ibm.com

Paul Chang Senior Consultant, Data Scientist, IBM Cloud paulyc@ tw.ibm.com 2 no AI without IA 3 3 3 3 3 3 AI Machine Learning Analytics Data The AI Ladder 3 Most are here Data Driven Insight Driven Digital