Dell IT Proven Dell IT s Big Data Journey with Big Possibilities

Size: px

Start display at page:

Download "Dell IT Proven Dell IT s Big Data Journey with Big Possibilities"

Gillian Booth
6 years ago
Views:

1 Dell IT Proven Dell IT s Big Data Journey with Big Possibilities Darryl Smith Chief Data Platform Architect, DE Patrick O Sullivan Sr Director, Analytics & BI

2 Agenda 1 Why are we here / current state challenges 2 How do we solve and evolve 3 Dell Data Lake Logical view Capabilities Physical View 4 Use cases 5 Key takeaways 2

3 Why are we here Two large entities (Dell & EMC) have come together to form the single largest privately held company Dell Technologies Data critical to success for the combined company is housed in multiple legacy systems and platforms. This requires extraordinary effort to harness the value of data to drive Dell s success. 3

Post merger challenges and business impacts Oracle ERP Business Impacts Legacy Dell Siebel CRM Order Mgmt Data Reservoir Actionable insights on customer Speed to market Master Data Missed cross-sell

4 Post merger challenges and business impacts Oracle ERP Business Impacts Legacy Dell Siebel CRM Order Mgmt Data Reservoir Actionable insights on customer Speed to market Master Data Missed cross-sell and up-sell opportunities Ability to integrate acquisitions SAP ERP Firewall Lack of agility, scalability and flexibility to develop new Analytical insights Legacy EMC SAP CRM Order Mgmt Master Data Firewall, network barrier IT Challenges Lack of Centralized Data Platform Overlapping tools System of Record Business logic transformation and filtering Lack of consolidated MDM Lack of consolidated information catalogue 4

5 How do we solve and evolve Develop a world class data ecosystem that provides the right data, in right place & at the right time to solve for current challenges and positions us for the Digital Transformation An insight-rich ecosystem to drive business value (few use cases): Improve Operational Efficiency Grow Revenue Improve Security and Compliance Improve CSAT Cross sell and upsell products Improve manufacturing quality Predictive device failure Maintenance and Support renewals Customer segmentation Reduce customer churn/dissatisfaction Sophisticated and personalized security 5

6 Dell data lake - logical architecture Sources ERP1 ERP2 CRM Chat Logs Machine Logs/IoT Complex Event Processing Pub-Sub Data Movement * Batch Micro-Batch Rules Based Decisions Services Tier * Streaming Analytics Microservices APIs Shared Data Platform * IT Managed/ Enterprise Integrated Processed Raw Data Quality Processing Business Managed Workspaces 3 Mobile Notifications Execution Tier * R/T Cache In-Memory NoSQL OLAP 5 Consumption Tier * Business Intelligence Reporting Ad Hoc Discovery Advanced Analytics Predictive Prescriptive Machine Learning Architecture Design Principles Platform to integrate multiple Systems of Record Ability to scale for volume, variety and velocity Enables real-time decisions & analytics Personalized and frictionless experience Continuous evolution to new and developing technology advancements Execution platforms for highperformance data services End to end self-service enablement Social Media Streaming 2 MDM 7 Data Warehouse 4 Applications Data Driven Mobile Analytical Defined path for innovation to production Showcase for Dell IT Proven 1 Metadata, Security & Governance 8 6 * Key components enabled for end-to-end self-service 6

Dell data lake - business view Sources ERP1 ERP2 CRM Chat Logs Machine Logs/IoT Social Media 1 Complex Event Processing Pub-Sub Data Movement * Batch Micro-Batch Streaming 2 Rules Based Decisions

7 Dell data lake - business view Sources ERP1 ERP2 CRM Chat Logs Machine Logs/IoT Social Media 1 Complex Event Processing Pub-Sub Data Movement * Batch Micro-Batch Streaming 2 Rules Based Decisions Services Tier * Streaming Analytics Microservices APIs Shared Data Platform * IT Managed/ Enterprise Integrated Processed Raw MDM Data Quality Processing Business Managed Workspaces Metadata, Security & Governance 3 7 Mobile Notifications Execution Tier * R/T Cache In-Memory NoSQL OLAP Data Warehouse Consumption Tier * Business Intelligence Reporting Ad Hoc Discovery Advanced Analytics Predictive Prescriptive Machine Learning Applications Data Driven Mobile Analytical 6 Personas ENTERPRISE ~ 50,000 BIZ ANALYST ~35,000 DATA ANALYST ~ 15,000 DATA SCIENTIST ~ 100 CONSUMER Millions * Key components enabled for end-to-end self-service 7

8 Dell data lake capabilities Enterprise Status Supports high availability (3x9), ultra highly available (4x9) and mission critical levels (5x9) Scalable/Elastic Data Governance Trusted data source Self-service Everything Enable the business to bring in additional data and perform analytics in an agile and frictionless manner Data ingestion Reporting & Analytics Data access through Governance Real-time Analytics Real-time dashboards Trigger and event handling Alert capabilities Analytics-driven business apps Complex event processing Data driven decision making Machine learning and AI embedded Predictive/statistical modeling Mobile and Desktop reporting/analytics access 8

9 Physical Architecture

Business Managed Workspaces Execution Tier * Micro-Batch MDM Metadata, Security &

10 Dell data lake physical architecture Sources Services Tier * Consumption Tier * Real-Time Batch Pub-Sub Data Movement * Shared Data Platform * IT Managed/ Enterprise Business Managed Workspaces Execution Tier * Micro-Batch MDM Metadata, Security & Governance HDFS ON ISILON HADOOP ON POWEREDGE w/scaleio VCE VBLOCK/VxRACK XTREMIO DATA DOMAIN 10

11 Dell Data Lake Use Cases

12 Overview of data science use cases Simplified Renewals Develop Insights around installed base renewals such as using CALM tool to improve renewal bookings User Behavior Analytics Organization-wide threat intelligence layer to predict anomalies when users access EMC Network. Used by CIRC Customer Support Service enabled dashboard with 360 data visualization of the health and wellness of customer s Dell EMC hardware & software Credits & Collections Predict customer s Patterns and Collection Behavior to improve revenue collection for GBS Services HR Voluntary Attrition 90 days early warning on employees with high flight risk Drive Failure Analytics Optimize Service Costs and user Experience by predicting disk health and failures (VNX, VMAX, XIO) Security & Compliance Grow Revenue Increase Efficiency Improve CSAT Improve enterprise and cloud security for Dell and customers via predictive models Identify new revenues using Predictive analytics Drive efficiency and productivity improvements using analytical insights Provide a better user experience with 360 insights 12

Increased efficiency Supply Chain Supply chain data integration is required to enable business to pursue cost synergies in MRP and materials planning, consolidating suppliers and processes in direct

13 Increased efficiency Supply Chain Supply chain data integration is required to enable business to pursue cost synergies in MRP and materials planning, consolidating suppliers and processes in direct and indirect procurement, measuring product quality and logistics metrics Solve for Large data sets such as MRP as they are too big for manual manipulation. Standardization of metrics across all users reporting views. Complex data views creating a consolidated demand view for parts or commodities could never be solved manually Need a highly reliable and flexible data transformation capability for Blending Critical Data together 13

Improve CSAT Customer Support MyService360 is an online capability that gives customers deep visibility into the health and wellness of their DellEMC environment, utilizing analytics, compelling

14 Improve CSAT Customer Support MyService360 is an online capability that gives customers deep visibility into the health and wellness of their DellEMC environment, utilizing analytics, compelling visualizations and multiple structured and unstructured data sources. Install Base Collected from enterprise systems and augmented with sensor data from installed systems. Code Level Analysis Collected from sensor data and analyzed against engineering data. Connectivity Status Status of hardware, based on sensor data. Health Risk Collected from a variety of sources and analyzed against technical & security bulletins, serviceable events and change orders. 14

Grow revenue Maintenance Renewals Customer Asset Lifecycle Management (CALM) solves the problem of Field and Renewals Sales not having timely access to accurate maintenance renewal quotes.

15 Grow revenue Maintenance Renewals Customer Asset Lifecycle Management (CALM) solves the problem of Field and Renewals Sales not having timely access to accurate maintenance renewal quotes. Contracts, Install Base, Price List, Contacts from Enterprise Systems Sensor data from Dell EMC hardware Analytics Data Scoring used to correct data inconsistencies and determine level of accuracy. Prescriptive Analytics to prioritize 15

Grow revenue CloudIQ CloudIQ is a no cost Software-as-a-Service (SaaS) offering that provides a simple monitoring interface for an unlimited number of Unity systems.

16 Grow revenue CloudIQ CloudIQ is a no cost Software-as-a-Service (SaaS) offering that provides a simple monitoring interface for an unlimited number of Unity systems. CloudIQ accesses data in near real time to enable monitoring and troubleshooting for Unity systems. Performance Key system performance metrics: Bandwidth, SPA Utilization, SPB Utilization, Block Latency and Used Capacity (IOPs for the most recent four hours) Capacity shows the usable, unconfigured, and free space available. (Efficiency Ratio for Thin LUNs, Snapshots, and Compression) Proactive Health Score identifies potential issues before they occur. (System Health, Configuration, Data Protection, Performance, and Capacity) Proactive Health Score Configuration Displays system configuration along with Pools, LUNS, and File Systems. 16

17 Use case benefits REAL DECISIONS ARE BEING MADE BASED OFF OF: Proven statistical models Actual data Real time reporting Benefits BETTER INSIGHTS LEAD TO BETTER OUTCOMES. Focus efforts in more profitable directions Proactive customer care creates customer/brand loyalty Consolidated view of customers allows more effective sales and marketing cycles MACHINE LEARNING ALLOWS: Improved products Proactive support Streamlined R&D 17

18 Key takeaways Don t try to solve all your problems with one platform Use the appropriate platform for the use case Execution tier MPP v. Hadoop Takeaways Using Data to make key decisions gives higher accuracy compared to gut feel Run Hadoop virtualized using ScaleIO for compute and Isilon for HDFS Separate compute from storage and grow them independently Rolling Upgrades with zero downtime Create sub clusters for managing licenses/subscriptions Run multiple distributions/versions on the same copy of data 18

19 19

20 Drilldown Slides

21 Sources 1 There are many sources of information, such as: core, run the business, applications, manufacturing and defect tracking, product testing results, service records, device sensor data, social media. Each source, taken independently, has valuable information, but merged together provides vastly more insights. ERPs MDMs Other internal apps Dell cloud-based apps External sources IOT streams API calls, e.g. Demandbase 21

Pub sub data movement 2 All enterprise data movement into, through, and out of the DDL will be pub-sub enabled Streaming, micro-batch, and batch data with eventing Under SLAs for refresh, quality,

22 Pub sub data movement 2 All enterprise data movement into, through, and out of the DDL will be pub-sub enabled Streaming, micro-batch, and batch data with eventing Under SLAs for refresh, quality, sharability, performance, etc. Provides data catalog and lineage information to governance Pub-Sub prevents vendor lock in Self-Service ingestion enabled via point-to-point or pub-sub framework With information governance and operational guardrails Path to IT production would require pub-sub, for with minimal co-dev rework Some ETL tools needed for transformation Avoid in-database procedural DML where possible Outbound data movement APIs Security Governed self-service Limited federated capabilities for querying external datasets 22

23 Shared data platform 3 Physical lake of data collected from various sources and co-located in a common platform Enterprise systems of record IoT Devices Dell and DellEMC products Social Media 3rd party sources Data can persist in various states of data integration Raw - Untransformed structured (same format as source system) and unstructured Processed - Refined, modeled, validated, and/or restructured to facilitate consumption Integrated - Structurally and semantically conformed across multiple sources and departments for shared consumption Data can persist in various structural forms Structured fixed schema Self-describing Unstructured Semi-Structured 23

24 Execution tier 4 Specialized platforms to fulfill consumption requirements that cannot be adequately met directly on Hadoop or MPP platforms Performance High concurrency of consumption Specialized functionality Transactional, Mission Critical High speed data services hub Ultra high concurrency or ultra low latency requirement Deployed to Public Cloud for agility or regional or compliance requirements Typically contains specially transformed or subsets of data in the Shared Data Platform Analytic engine for data in Shared Data Environment typically found in a data warehouse OLAP or Star Schema Existing Data Warehouses Volume of processing and data will organically shrink over time Growing data persistence in Shared Data Platform Migration of existing transformation and aggregation processes and BI to Shared Data Platform during course of project-driven modifications Implementation of new transformation and aggregation processes and BI on Shared Data Platform 24

25 Services tier 5 Tools and frameworks used to provide services to other applications Mobile push notifications In-line data quality processing and threshold alerting Microservices APIs Streaming analytics Rules-based decisioning Complex event processing Built-in security at the Service layer 25

26 Consumption tier 6 Tools and frameworks used to provide reporting and analytical views to the end-consumer Examples Business Objects Tableau Pivotal Cloud Foundry Datameer Native Hadoop tools: R, Python, Spark, MadLib, Generally used for Enterprise/ Operational reporting Ad hoc BI and OLAP Data science Data mining Machine learning Discovery analytics Analytics Descriptive, Diagnostic, Predictive, Prescriptive Downstream feeds - Internal and external 26

27 MDM 7 Master Data Management(MDM) is the process, governance and tools that consistently define and manage the critical data of an organization, to provide a single point of reference. Data Quality provides capability to measure and monitor critical data elements to ensure compliance with data standards Current Master Data domains Customer Product Install base 27

28 Metadata, security & governance 8 Metadata: Catalog for technical metadata in the DDL with the business glossary and relationships Data Lineage connecting source system of record through ingestion and transformation Security: Single role based access model throughout the DDL - managed through native security capabilities of the shared data platform technologies Encryption for data in motion and data at rest Governance: Enterprise Data Governance Center consists of data owners/stewards responsible Managing and granting access to the data Defining business glossary and relationships Ensuring data quality 28

29 Dell data lake - data flow architecture CONSUMPTION Application and Analytical Workspaces APPLICATION and ANALYTICAL WORKSPACES Mobile Tableau/BOBJ Collaborate Colibra/Peer 2 Peer Analyze R/SAS/Python/Spark Enterprise Mission Critical EXECUTION CASSANDRA XD GEMFIRE TD ASTER POSTGRESQL MEMSQL CLOUDERA GREENPLUM DB TD TERADATA DATA/ANALY TICS HDFS/MPP INGESTION SpringXD Self-Service Infomatica Social Media Sensor Network UNSTRUCTURED Web Supplier Transactional Systems Market ERP CRM PLM STRUCTURED 29

30 Key business outcomes Identify new revenues using predictive analytics Identify new revenues using predictive analytics Drive efficiency and productivity improvement using analytical insights 30

EMC IT Big Data Analytics Journey. Mahmoud Ghanem Sr. Systems Engineer

EMC IT Big Data Analytics Journey Mahmoud Ghanem Sr. Systems Engineer Agenda 1 2 3 4 5 Introduction To Big Data EMC IT Big Data Journey Marketing Science Lab Use Case Technical Benefits Lessons Learned