IBM Big Data & Analytics Reference Architecture V1

Transcription

1 Date: June 12 th, 2014 IBM Big & Analytics Reference Architecture V1 Overview

2 Building an Architecture for Big & Analytics problems 2

3 Purpose This presentation, derived from the IBM Big & Analytics Reference Architecture document is meant to drive consistency across targeted technical sales channels in the roll-out of the IM & BA reference architectures aligned to the 5 big data use cases and the industry scenarios. The purpose of IBM Big & Analytics (BD&A) Reference Architecture is to inform and guide sales, technical sales, services and other professionals who are involved in selling IBM solutions or deploying them with clients. The Reference Architecture is intended to be used by a wide range of professionals selling IBM software and designing end-to-end big data & analytics client solutions. The reference architecture can also be reused by IBM clients and partners to gain further insight on how to benefit from IBM Software in big data & analytics projects (appropriate disclosure is required). The target audience also depends on the specific work product included in the reference architecture. The Architecture Overview, Business Directions. Requirements, Technology Brief are expected to be consumed by business leaders and architects. 3

4 The BDA RA Reference Architecture material is based on TeamSD and applies to the en9re sales cycle Understand Client Define Client Requirements Design Solu0on Detail Design to Define BOM Best Prac0ces DevOps Business Drivers Use Cases System Context Architecture Decisions Best Prac9ces Describe the key business drivers for the project, the KPIs or CSFs, and how they align with Cloud compuang. What are the funcaonal requirements expected from the Cloud and who are the key actors. Expressed as Use Cases. The system context should define the boundary of the Cloud, and the integraaons with OSS / BSS systems Clearly documented decisions on key architectural points including the raaonale for the decision. Define the overall Ameline, phases, and key milestones that will shape the plan and overall delivery. Func9onal & Non- Func9onal Requirements Architecture Overview Opera9onal Model Dev Ops 4 Content provided by BDA RA Business Drivers Use Cases Requirements, Security Architecture Overview, Architecture Decisions Operational Model, DevOps Best Practices NFRs should be defined to cover the volumes, capacity, scale, availability, security, operaaonal and monitoring aspects of the Cloud Architecture overview diagram should define the high level components, their placement. Technology Brief Define the boundaries of the project, inclusions, exclusions, dependencies, and align phases with milestones in the roadmap. Design and consider the components of the soluaon both at a physical and logical level. Design and consider the components of the soluaon both at a physical and logical level.

5 The IBM Business Analytics and Optimization: Enterprise Architecture View 5

6 The IBM Big & Analytics Reference Architecture: High Level Capabilities The IBM Business Analytics and Optimization: Detailed Capability View 6

7 The IBM Business Analytics and Optimization: Logical View The IBM Big & Analytics Reference Architecture: Software Product Mapping View 7

8 The BDA RA Reference Architecture material is based on TeamSD and applies to the en9re sales cycle Understand Client Define Client Requirements Design Solu0on Detail Design to Define BOM Best Prac0ces DevOps Business Drivers Use Cases System Context Architecture Decisions Best Prac9ces Describe the key business drivers for the project, the KPIs or CSFs, and how they align with Cloud compuang. What are the funcaonal requirements expected from the Cloud and who are the key actors. Expressed as Use Cases. The system context should define the boundary of the Cloud, and the integraaons with OSS / BSS systems Clearly documented decisions on key architectural points including the raaonale for the decision. Define the overall Ameline, phases, and key milestones that will shape the plan and overall delivery. Func9onal & Non- Func9onal Requirements Architecture Overview Opera9onal Model Dev Ops 8 Content provided by BDA RA Business Drivers Use Cases Requirements, Security Architecture Overview, Architecture Decisions Operational Model, DevOps Best Practices NFRs should be defined to cover the volumes, capacity, scale, availability, security, operaaonal and monitoring aspects of the Cloud Architecture overview diagram should define the high level components, their placement. Technology Brief Define the boundaries of the project, inclusions, exclusions, dependencies, and align phases with milestones in the roadmap. Design and consider the components of the soluaon both at a physical and logical level. Design and consider the components of the soluaon both at a physical and logical level.

9 Business Drivers Business Drivers Informa9on and data pervades every aspect of the organiza9on from cost containment to revenue growth to customer interac9on and marke9ng to regulatory repor9ng to risk repor9ng and metrics of profitability as well as your organiza9on liquidity constraints. Therefore informa9on management architecture and managing data is a mul9- disciplinary task. Selfservice Realtime Integration Batch Security Event Global Process Interaction How do you get there and deliver for the client? 9

10 Business Drivers Business Drivers: Big introduces new opportunities and concepts Scale: Decision Frequency: at Rest: Up to exabytes. Up to 10,000 times larger than traditional data warehouses Up to real time using streaming data patterns is persisted in a physical medium and is considered relatively static in Motion: flowing constantly through a network or other data transport mechanism. persisted in memory on a temporary basis is also considered to be in motion Being able to manage and leverage these new dimensions creates a unique opportunity to change the game for our clients 10

11 Business Drivers Business Driver: Why Big & Analytics are Important? is emerging as the world s newest resource for competitive advantage The power of all data coming together structured & unstructured data (sensor, social, logs, machine, etc) with the power of New Technology Real- Time Analytics Delivering Actionable Insights v Uncover new insights to transform the business Why did it happen? What is likely to happen? What s the best course of action based on what you ve learned? v Empower many more employees within the organization At every level of the organization to make better decisions v Improve effectiveness & competitiveness of the business Make speed a differentiator Monetize the data itself Be more right, more often v Manage risk Protect against poor decision-making (riskopportunity equation right) Protect against security and privacy risks 11 Hadoop, Streaming, Cognitive, In-Memory, Exploration v New and more effective approach to perform analytics Move analytics to the data, ( in-motion & native format) Leverage open source and commodity hardware (cost savings)

12 Use Cases Use Cases: Every Industry can Leverage Big and Analytics Banking Insurance Telco Energy & Utilities Media & Entertain Optimizing Offers and Cross-sell Customer Service and Call Center Efficiency 360 View of Domain or Subject Catastrophe Modeling Fraud & Abuse Pro-active Call Center Network Analytics Location Based Services Smart Meter Analytics Distribution Load Forecasting/ Scheduling Condition Based Maintenance Business process transformation Audience & Marketing Optimization Retail Travel & Transport Consumer Products Govern. Healthcare Actionable Customer Insight Merchandise Optimization Dynamic Pricing Customer Analytics & Loyalty Marketing Predictive Maintenance Analytics Shelf Availability Promotional Spend Optimization Merchandising Compliance Civilian Services Defense & Intelligence Tax & Treasury Services Measure & Act on Population Health Outcomes Engage Consumers in their Healthcare Automotive Chemical & Petroleum Aerospace & Defense Electronics Life Sciences Advanced Condition Monitoring Warehouse Optimization Operational Surveillance, Analysis & Optimization Warehouse Consolidation, Integration & Augmentation Uniform Information Access Platform Warehouse Optimization Customer/ Channel Analytics Advanced Condition Monitoring Increase visibility into drug safety and effectiveness 12

13 Use Cases: The 5 Key Use Cases for Big Use Cases Big Exploration Find, visualize, understand all big data to improve decision making Enhanced 360 o View of the Customer Extend existing customer views (MDM, CRM, etc) by incorporating additional internal and external information sources Security/ Intelligence Extension Lower risk, detect fraud and monitor cyber security in realtime 13 Operations Analysis Analyze a variety of machine data for improved business results Warehouse Augmentation Integrate big data and data warehouse capabilities to increase operational efficiency

14 The BDA RA Reference Architecture material is based on TeamSD and applies to the en9re sales cycle Understand Client Define Client Requirements Design Solu0on Detail Design to Define BOM Best Prac0ces DevOps Business Drivers Use Cases System Context Architecture Decisions Best Prac9ces Describe the key business drivers for the project, the KPIs or CSFs, and how they align with Cloud compuang. What are the funcaonal requirements expected from the Cloud and who are the key actors. Expressed as Use Cases. The system context should define the boundary of the Cloud, and the integraaons with OSS / BSS systems Clearly documented decisions on key architectural points including the raaonale for the decision. Define the overall Ameline, phases, and key milestones that will shape the plan and overall delivery. Func9onal & Non- Func9onal Requirements Architecture Overview Opera9onal Model Dev Ops 14 Content provided by BDA RA Business Drivers Use Cases Requirements, Security Architecture Overview, Architecture Decisions Operational Model, DevOps Best Practices NFRs should be defined to cover the volumes, capacity, scale, availability, security, operaaonal and monitoring aspects of the Cloud Architecture overview diagram should define the high level components, their placement. Technology Brief Define the boundaries of the project, inclusions, exclusions, dependencies, and align phases with milestones in the roadmap. Design and consider the components of the soluaon both at a physical and logical level. Design and consider the components of the soluaon both at a physical and logical level.

15 Functional Requirements Functional requirements PROJECT DEFINITION USE CASES identify FUNCTIONAL REQUIREMENTS 15

16 Functional requirements Functional Requirements: Purpose Capture common functional requirements Provide consistent, reusable, and proven approach Can be adapted for customer deliverables Readiness Assessments Customer Proposals 16

17 Functional requirements How to make use of the functional requirements Acquisition Real-Time Processing & Analytics Integration Analytics Repositories Shared Operational Information Information Access Information Interaction Governance Security & Business Continuity Infrastructure 17

18 Non Functional Requirement: Security Non- Functional requirements 18

19 Security Non- Functional requirements Goals: Prevent, detect and address system breaches Mitigate internal and external threats Ensure integrity & privacy of sensitive data Ensure the availability of data Reduce cost of compliance Protect at Rest & In Motion InfoSphere Guardium Activity Monitoring Vulnerability Assessments InfoSphere Optium Masking InfoSphere Guardium Encryption Expert Key Lifecycle Manager Access Controls Access Manager family Federated Identity Manager Identity Manager/Role Lifecycle Manager Fine Grained entitlements MDM and GNR Security Policy Manager Privileged Identity Manager 19

20 Non- Functional requirements Security I2 Analyst s Notebook Connectors Network Telemetry Monitoring Appliance (Optional) Real-time Ingest & Processing InfoSphere Streams Video/audio Network Geospatial Predictive Big Storage & Analytics Warehouse Criminal Information Tracking System Surveillance Monitoring System Connectors InfoSphere BigInsights Text and entity analytics mining Machine learning Deep analytics Operational analytics Large scale structured data management Security Info & Event Management (SIEM) Unstructured & Streaming Structured 20

21 The BDA RA Reference Architecture material is based on TeamSD and applies to the en9re sales cycle Understand Client Define Client Requirements Design Solu0on Detail Design to Define BOM Best Prac0ces DevOps Business Drivers Use Cases System Context Architecture Decisions Best Prac9ces Describe the key business drivers for the project, the KPIs or CSFs, and how they align with Cloud compuang. What are the funcaonal requirements expected from the Cloud and who are the key actors. Expressed as Use Cases. The system context should define the boundary of the Cloud, and the integraaons with OSS / BSS systems Clearly documented decisions on key architectural points including the raaonale for the decision. Define the overall Ameline, phases, and key milestones that will shape the plan and overall delivery. Func9onal & Non- Func9onal Requirements Architecture Overview Opera9onal Model Dev Ops 21 Content provided by BDA RA Business Drivers Use Cases Requirements, Security Architecture Overview, Architecture Decisions Operational Model, DevOps Best Practices NFRs should be defined to cover the volumes, capacity, scale, availability, security, operaaonal and monitoring aspects of the Cloud Architecture overview diagram should define the high level components, their placement. Technology Brief Define the boundaries of the project, inclusions, exclusions, dependencies, and align phases with milestones in the roadmap. Design and consider the components of the soluaon both at a physical and logical level. Design and consider the components of the soluaon both at a physical and logical level.

22 Architecture Overview: and Analytics Architecture Overview 22

23 Architecture Overview The IBM Business Analytics and Optimization Enterprise Architecture View 23

24 IBM Big and Analytics Reference Architecture High Level Capabilities Big & Analytics Sources New Sources Tradi9onal Sources AcquisiAon ( Extract, Replicate, Copy) & ApplicaAon Access Integration Streaming Computing Real-Time Analytical Processing Analytical Sources Hadoop & Exploration Warehousing Shared Operational Information Information Access Semantic Layer Delivery & Visualization Actionable Insight Decision Management Reporting, Analysis & Content Analytics Discovery & Exploration Predictive Analytics & Modeling 24 Governance Event Detec9on and Ac9on Security & Business Con9nuity Management PlaQorms

25 IBM Big and Analytics Reference Architecture - Logical Overview Big & Analytics Sources Streaming Compu9ng Ac9onable Insight Enhanced Applica9ons New Sources Real- Time Analy9cal Processing Decision Management Customer Experience Machine & Sensor Image & Video Enterprise Content Social Internet Tradi9onal Sources Third-Party Acquisi9on & Applica9on Access Integra9on Quality, TransformaAon & Load Landing Explora9on & Archive Big Repository Analy9cal Sources Integrated Warehouse Enterprise Warehouse Deep Analy9cs & Modeling Analy9cal Appliances Interac9ve Analysis & Repor9ng Marts Discovery & ExploraAon Modeling & PredicAve AnalyAcs Analysis & ReporAng Planning & ForecasAng New Business Model Financial Performance Risk OperaAons & Fraud Transactional Application Master & Reference Shared Opera9onal Informa9on Content Hub Activity Hub Metadata Catalog Content AnalyAcs IT Economics Governance Event Detec9on and Ac9on Security & Business Con9nuity Management 25 PlaQorms

26 IBM Big and Analytics Reference Architecture Detailed Capabilities Big & Analytics Sources New Sources Machine & Sensor Image & Video Enterprise Content Social Internet Tradi9onal Sources Third-Party Transactional Application AcquisiAon ( Extract, Replicate, Copy) & ApplicaAon Access Integration Integration Ingestion Extract & Subscribe Initial Stage Quality Clean Staging Transformation Load Ready Load Master & Reference Streaming Computing Real-Time Analytical Processing Predictive Analytics Shared Operational Information Content Hub Analytical Sources Hadoop & Exploration Landing Archive Warehousing Warehouse Marts ODS Time Persistent Activity Hub Real-Time Insights Exploration Indexing Accelerators (In-Memory) Analytical Appliances Cubes Sandbox Metadata Catalog Information Access Semantic Layer Delivery & Visualization Services Publish Access Caching Delivery Virtualization Federation Link Actionable Insight Decision Management Rules Management Reporting, Analysis & Content Analytics Planning, Forecasting Budgeting Reporting Collaboration Real Time Decision Management Scorecards, Dashboards Query & Analysis Storytelling Alerting, Monitoring Discovery & Exploration Annotation Search Predictive Analytics & Modeling Predictive Analytics Simulation Mining Text Analytics Optimization Correlations 26 Corporate Governance Information Governance Governance Event Detec9on and Ac9on Security & Business Con9nuity Management Private Clouds Public Clouds Appliances Custom HW Solutions PlaQorms

27 Big & Analytics Component Model Real-Time Analytical Processing Components Applica9on Development Framework Administra9on and Monitoring Services Delivery Services Enterprise Sources Real- Time Analy9cal Processing Opera9onal Applica9ons Machine & Sensor Image & video Integra9on Services Predic9ve Analy9cs Services Decision Management Services Log Sensor & Capture Services Streaming Pipeline Applica9on Integra9on Services Enterprise Content Internet Transactional Big Repository (Hadoop) Analy9cal Sources Warehouse & Marts 27

28 Big & Analytics Component Iteration Real-Time Analytics Process Component Interaction Applications & Sources Streaming Computing Analytical Sources (Repositories) Transactions, Applications & Devices Internet Of Things External ApplicaAons New Sources Machine Sensor Images Videos Content Social Internet Sensor & Capture Real-Time Analytical Processing Real- Time Integra9on Predic9ve Analy9cs (Real- Time Scoring) Build History Analytical Sources (repositories) Big Repository (Hadoop) Interac9ve Analysis & Repor9ng ( Marts) Warehouse Business Analysts Front- Office ApplicaAons Back- Office ApplicaAons Tradi9onal Sources Third-Party Transactional Application.. Model Feedback Loop Analy9cal Appliances (Predic9ve modeling) Quants & Scien9st Real- Time Decision Management Monitoring & Aler9ng Applica9on Integra9on Real-Time Monitoring, Alerting & event Handling 28

29 The BDA RA Reference Architecture material is based on TeamSD and applies to the en9re sales cycle Understand Client Define Client Requirements Design Solu0on Detail Design to Define BOM Best Prac0ces DevOps Business Drivers Use Cases System Context Architecture Decisions Best Prac9ces Describe the key business drivers for the project, the KPIs or CSFs, and how they align with Cloud compuang. What are the funcaonal requirements expected from the Cloud and who are the key actors. Expressed as Use Cases. The system context should define the boundary of the Cloud, and the integraaons with OSS / BSS systems Clearly documented decisions on key architectural points including the raaonale for the decision. Define the overall Ameline, phases, and key milestones that will shape the plan and overall delivery. Func9onal & Non- Func9onal Requirements Architecture Overview Opera9onal Model Dev Ops 29 Content provided by BDA RA Business Drivers Use Cases Requirements, Security Architecture Overview, Architecture Decisions Operational Model, DevOps Best Practices NFRs should be defined to cover the volumes, capacity, scale, availability, security, operaaonal and monitoring aspects of the Cloud Architecture overview diagram should define the high level components, their placement. Technology Brief Define the boundaries of the project, inclusions, exclusions, dependencies, and align phases with milestones in the roadmap. Design and consider the components of the soluaon both at a physical and logical level. Design and consider the components of the soluaon both at a physical and logical level.

30 Technology Brief Technology Brief 30

31 Technology Brief IBM Big and Analytics Reference Architecture - Software Product View Sources New Sources Machine & Sensor Image & Video Enterprise Content Social Internet Tradi9onal Sources Third-Party Transactional Application Acquisi9on & Applica9on Access Integra9on InfoSphere Federa9on Server InfoSphere Informa9on Server InfoSphere Replica9on InfoSphere Click InfoSphere Master Management Streaming Compu9ng Real-Time Analytical Processing Landing, Archive & Explora9on Hadoop System Pure for Hadoop (PDH) InfoSphere BigInsights InfoSphere Streams Analytical Sources Integrated Warehouse Pure for Opera9onal Analy9cs (PDOA) Industry Models InfoSphere Warehouse Packs Shared Opera9onal Informa9on Enterprise Content Management Deep Analy9cs & Modeling Pure for Analy9cs (PDA) Repor9ng & Analysis Pure for Analy9cs (PDA) BLU Accelera9on Metadata Workbench Ac9onable Insight Decision Management ILOG ODM SPSS ADM Discovery & Explora9on SPSS Analy9c Watson Catalyst Explorer Watson Analy9cs Planning & Forecas9ng Concert Cognos TM1 InfoSphere Bus. Info. Exchange Predic9ve Analy9cs & Modeling SPSS Sta9s9cs Cognos Insights Cognos Express Content Analy9cs Watson Content Analy9cs SPSS Modeller Analysis & Repor9ng Cognos BI Cognos BI PaYern Cognos Express Cognos BI PaYern w/ BLU Cognos Social Media Analy9cs 31 InfoSphere Business Informa9on Exchange Governance Guardium Op9m Guardium Event Detec9on and Ac9on Security & Business Con9nuity Management PlaQorms So\layer IBM Cloud Power Systems Z Systems Pure Systems Tivoli

32 The BDA RA Reference Architecture material is based on TeamSD and applies to the en9re sales cycle Understand Client Define Client Requirements Design Solu0on Detail Design to Define BOM Best Prac0ces DevOps Business Drivers Use Cases System Context Architecture Decisions Best Prac9ces Describe the key business drivers for the project, the KPIs or CSFs, and how they align with Cloud compuang. What are the funcaonal requirements expected from the Cloud and who are the key actors. Expressed as Use Cases. The system context should define the boundary of the Cloud, and the integraaons with OSS / BSS systems Clearly documented decisions on key architectural points including the raaonale for the decision. Define the overall Ameline, phases, and key milestones that will shape the plan and overall delivery. Func9onal & Non- Func9onal Requirements Architecture Overview Opera9onal Model Dev Ops 32 Content provided by BDA RA Business Drivers Use Cases Requirements Architecture Overview, Architecture Decisions Operational Model, Security guidance DevOps Best Practices NFRs should be defined to cover the volumes, capacity, scale, availability, security, operaaonal and monitoring aspects of the Cloud Architecture overview diagram should define the high level components, their placement. Technology Brief Define the boundaries of the project, inclusions, exclusions, dependencies, and align phases with milestones in the roadmap. Design and consider the components of the soluaon both at a physical and logical level. Design and consider the components of the soluaon both at a physical and logical level.

33 Architecture Decisions Guidance Big Analytics key Architecture Decision areas 33

34 Architecture Decisions Guidance ADC001 Storing large volumes of data for analytics 34

35 Architecture Decisions Guidance ADC002 High Performance Warehousing & Analytics 35

36 Architecture Decisions Guidance AD-C006: Integration of structured & unstructured content 36

37 Architecture Decisions Guidance Architecture Guidance Patterns For patterns Re-usable Patterns for Big

38 Legend Mart Virtual Mart Elements Architecture Decisions Guidance Business Intelligence (BI) information that must typically be aligned to new acquired HDFS sources.. Accessible via In-Memory, Hybrid or Traditional databases. Virtual sources that retrieve information from underlying big data source on-the fly. stays in-place in HDFS. 38 Big Table Big Index Big Warehouse (HDFS) Application Process Spreadsheet-like row and column data extracted from an HDFS. Typically augmented and extended using row/column metaphor. Ad hoc dimensional views as popular. Value Indexed information used by search engines. Column indexing allows faceted dimensions to be explored. that relies on a Massively Parallel Processing (MPP) environment to acquire, transform and supply information to applications. The Apache Hadoop processing environment and associated Hadoop Distributed File System (HDFS) accessed with NoSQL queries is the a de facto instantiation of this storage class. Anything that augments, transforms, extracts or otherwise processes that data. Can be simple like a REST or Native API function. Also includes applications related Information Interaction. 38

39 Architecture Decisions Guidance Solution Pattern 1: Landing Zone Warehouse Pattern 3. Applications Associated Use Cases Enhanced 360 View of the Customer, Warehouse Augmentation Use Description 2. Big Report Mart Reporting and analysis in finance, GRC or similar domain. extracted from Landing Zone Extract 1. Extrac t Source : Social, Machine/RFID, Transactions Big Warehouse (Landing Zone) Typical Steps 1. Big Warehouse HDFS Landing Zone built via ETL batch processes. Mix of structured, semi-structured and unstructured data extracted from a variety of external sources. 2. Big Report Mart loaded via batch ETL. Structure matches modeled BA elements. Options: In-Memory, Dynamic Cube, Traditional. 3. Information Interaction using SQL / MDX queries

40 Architecture Decisions Guidance Solution Pattern 2: Landing Zone Virtual Warehouse Pattern Associated Use Cases 3. Applications Enhanced 360 View of the Customer, Warehouse Augmentation Use Description 2. Virtual Tables Reporting and analysis in finance, GRC or similar domain. Uset stays in Landing Zone 1. Extrac t Source : Social, Machine/RFID, Transactions Big Warehouse (Landing Zone) Typical Steps 1. Big Warehouse HDFS (a.k.a. Landing Zone) built via ETL batch processes. Mix of structured, semistructured and unstructured data extracted from a variety of external sources. 2. Virtual Report Mart Virtual database built using SQL/ Hive Style over Big Warehouse HDFS. stays in HDFS. 3. Information Interaction using SQL/MDX queries

41 Architecture Decisions Guidance Solution Pattern 3: Landing Zone Table Report Mart Pattern 4. Applications Associated Use Cases Big Exploration, Security / Intelligence Extension Use 3. Big Summary Mart Description Reporting and analysis in finance, GRC or similar domain. 2. Export Big Table Ad hoc exploration and discovery performed use Big Table exploration tool (like Big Sheets) Extract Typical Steps 1. Extrac t Source : Social, Machine/RFID, Transactions Big Warehouse (Landing Zone) 1. Big Warehouse HDFS Landing Zone built via ETL batch processes. Mix of structured, semi-structured and unstructured data extracted from a variety of external sources. 2. Big Table built from landing zone sources. 3. Big Summary Mart Virtual database built from Big Table Content.. 4. Information Interaction using SQL/MDX queries

42 Architecture Decisions Guidance Component Pattern 7.1: Analytics Mart Pattern Information Interaction Applications Query from Shared Analytics Information Zone Characteristics Analytics Mart SaaS Search SQL/MDX Query and Discovery NoSQL Presentation, Visualization, Sharing Query, Extract SaaS Search Shared Analytics Information Zone Semantic Information from Shared Analytics Information Zone SaaS Search SQL/MDX Sub-pattern variant of Component Pattern 7: Exploration Mart Pattern where exploration is moved to the Information Interaction Domain expert (Line Of Business) constructs BA Analysis data using variety of tools and sources. Components 1. Analytics Mart built from Discovery Layer Landing Area Zone, Sources and other Sources. 2. Query and Discovery tools aid with search, extract and disambiguation of uncertain sources and relationships. 3. Presentation, Visualization Sharing optionally performed over Analytics Mart 4. Analytics Mart optionally shared to other components in Information Interaction. Discovery Mart Big Warehous e Source and Events Other Corporate & Web Sources Attributes Type Size Update Interval Rates Quality Response Time 42 Report Mart (In-Memory) Tabular In-Memory Up to 10 TB Variable Medium Variable Fast 42

43 The BDA RA Reference Architecture material is based on TeamSD and applies to the en9re sales cycle Understand Client Define Client Requirements Design Solu0on Detail Design to Define BOM Best Prac0ces DevOps Business Drivers Use Cases System Context Architecture Decisions Best Prac9ces Describe the key business drivers for the project, the KPIs or CSFs, and how they align with Cloud compuang. What are the funcaonal requirements expected from the Cloud and who are the key actors. Expressed as Use Cases. The system context should define the boundary of the Cloud, and the integraaons with OSS / BSS systems Clearly documented decisions on key architectural points including the raaonale for the decision. Define the overall Ameline, phases, and key milestones that will shape the plan and overall delivery. Func9onal & Non- Func9onal Requirements Architecture Overview Opera9onal Model Dev Ops 43 Content provided by BDA RA Business Drivers Use Cases Requirements Architecture Overview, Architecture Decisions Operational Model, Security guidance DevOps Best Practices NFRs should be defined to cover the volumes, capacity, scale, availability, security, operaaonal and monitoring aspects of the Cloud Architecture overview diagram should define the high level components, their placement. Technology Brief Define the boundaries of the project, inclusions, exclusions, dependencies, and align phases with milestones in the roadmap. Design and consider the components of the soluaon both at a physical and logical level. Design and consider the components of the soluaon both at a physical and logical level.

44 Operational Model Operational Model The operational model introduces the high-level operational nodes and associated deployed software components of the Big Enhanced Analytics System Architecture. In early stages Operational Model is used: As an early basis for design reviews and walkthroughs, including confirmation that the business problem is well articulated and that there is a viable IT solution. As a way of dividing large problems so that each node can be worked on in relative isolation, but yet be part of the same solution vision As the basis for early analysis of nonfunctional requirements such as performance, availability, and capacity, including confirmation of the viability of a solution through specification of the expected nonfunctional characteristics of nodes and components. To identify necessary technical, infrastructure, and other middleware components and subsystems. To contribute to early estimates of the cost of the infrastructure to be used both for budgeting and as part of the business case for the solution. In the later stage, an Operational Model at the specification level is used: To document the distribution of application and technical subsystems (deployment units) on preliminary (conceptual or specified) nodes so they can ultimately be installed and run on physical computer systems and on virtualized environments. As the basis for detailed design reviews and walkthroughs, prior to finalizing product selection. As a detailed technical specification against which an architect can evaluate alternative products or even against which technology vendors can submit tenders. As the basis for detailed prediction of performance, availability, and other service level characteristics. (Predictions are based on the overall architecture and the specifications of deployment units within it. It will have to be revisited, via system tests, when specific products have been chosen.). As the basis for a check that all the necessary business and technical functionality has been identified. As the basis for cost estimates of the required infrastructure. 44

45 Operational Model Continued Operational Model External Environment DMZ Internal Network Big and Analytics Zone Corporate Domain Zone Sources InfoSphere Streams Node Pure System for Operational Analytics Node InfoSphere Explorer Node Public & Private Networks Protocol Firewall Node Edge Server Node Reverse Proxy Node Domain Firewall Node Infosphere BigInsights Node DB2 Informix Node InfoSphere MDM Node InfoSphere Guardium Node InfoSphere Server Node IBM Watson Analytics InfoSphere Optim Node Cognos BI Node Cognos TM1 Node SPSS Modeler Server Node SPSS Analytic Server Node SPSS Statistic Server Node SPSS C&D Services Node Enterprise Firewall Node Enhanced Applications Node Customer Existing Enterprise Systems Infrastructure Management and Security The Operational Model is based on the Big Enhanced Analytics architecture overview diagram. This Operational Model represents the Big Enhanced Analytics solution conceptual topology and it is showing key nodes across IT zones. This is only conceptual operational representation. 45

46 Operational Model Big & Analytics Operational Model Continued Internal Network Big and Analytics Zone External Environment DMZ bases cluster DB2/Informix IBM Big and Analytics PowerEdge T310 PowerEdge T310 VMVM VM VMVM VMVM VM 2 VMware virtualization 4 System x3250 System x3250 VM VM HTTP/HTTPS Virtualized Server Cluster Domain Firewall Node F5 Load Balancer Protocol Firewall Node Public & Private Networks IBM Infosphere BigInsights Hadoop Cluster IBM Big and Analytics System x3250 System x3250 System x3250 System x3250 System x3250 System x3250 System x3250 System x3250 System x3250 System x3250 System x3250 System x3250 System x3250 System x3250 System x3250 System x3250 System x3250 System x3250 System x3250 VMVM VM VMVM VMVM VM 2 System x3250 Rack 1 Rack 2 VMware virtualization 4 Infrastructure Management and Security Pure System for Analytics VMVM VM VMVM VMVM VM VMware virtualization Customer Existing Enterprise Systems PowerEdge T310 4 PowerEdge T310 PowerEdge T This Operational Model represents the Big Enhanced Analytics deployment topology. In this example, the Dell blades and VMware virtualized environment are being used for the majority of software components for deployments. bases are being deployed into Dell servers. The Hadoop Cluster is deployed into IBM two Rack Systems and Warehouse and Analytics are deployed into Pura Systems for Analytics.

47 The BDA RA Reference Architecture material is based on TeamSD and applies to the en9re sales cycle Understand Client Define Client Requirements Design Solu0on Detail Design to Define BOM Best Prac0ces DevOps Business Drivers Use Cases System Context Architecture Decisions Best Prac9ces Describe the key business drivers for the project, the KPIs or CSFs, and how they align with Cloud compuang. What are the funcaonal requirements expected from the Cloud and who are the key actors. Expressed as Use Cases. The system context should define the boundary of the Cloud, and the integraaons with OSS / BSS systems Clearly documented decisions on key architectural points including the raaonale for the decision. Define the overall Ameline, phases, and key milestones that will shape the plan and overall delivery. Func9onal & Non- Func9onal Requirements Architecture Overview Opera9onal Model Dev Ops 47 Content provided by BDA RA Business Drivers Use Cases Requirements, Security Architecture Overview, Architecture Decisions Operational Model, DevOps Best Practices NFRs should be defined to cover the volumes, capacity, scale, availability, security, operaaonal and monitoring aspects of the Cloud Architecture overview diagram should define the high level components, their placement. Technology Brief Define the boundaries of the project, inclusions, exclusions, dependencies, and align phases with milestones in the roadmap. Design and consider the components of the soluaon both at a physical and logical level. Design and consider the components of the soluaon both at a physical and logical level.

48 Best Practices Big Analytics Implementation best practices Identify data components Identify the Implementation pattern(s) Enterprise Content Management Warehousing, Management ; Business Intelligence; Analytics data at rest & data in motion, modeling, Governance Big data exploration: Enhanced 360-degree view of the customer: Security/intelligence extension: Operations analysis: warehouse augmentation: Develop a maturity Model Develop a Big capability maturity model that will reflect your enterprise business and IT needs. Refer to other models. Ex. IBM Governance Council Maturity Model à Develop/use BDA Reference architecture Develop a Big Analytics Reference architecture that serves as a blueprint for your Big Analytics solutions. 48

49 Big Analytics Deployment best practices Best Practices Identify deployment options BYO Hardware IT Appliances: Private Cloud: Public Cloud: Identify security requirements People,, Application and infrastructure security needs; Security intelligence needs Risk management Develop a BDA Governance Monitor big data activity from applications and users for threats. Establish data traceability and auditability; Enforce change controls ; Encrypt and mask data to make it unusable Information is understood Information is correct Information is holistic Information is current Information is secure. Information is documented 49

50 Big Analytics maturity model Best Practices Ad-hoc; inconsistent; some PoCs / trials; stand alone Big Analytics environment planned & managed project specific implementation; Some hadoop skills; some insights with EDW 3. Defined Quantitative defined objectives; Predictable outcomes; End-to-end integration with EDW; 4. Quantitatively Managed Big organizational governance; continuous updates inline with business objectives 5. Optimizing 2. Managed 1. Initial 50

51 Best Practices Big and Analytics - Information governance 51

52 Best Practices Big & Analytics - Security intelligence example Requirement 1: Enhanced Intelligence and Surveillance Insight Requirement 2: Real-time Cyber Attack Prediction and Mitigation Requirement 3: Crime Prediction and Protection 52

53 Best Practices Security intelligence example implementation steps Identify data components Business Intelligence; Analytics data at rest & data in motion, modeling, Governance Analyze data-in-motion and at rest Analyze network traffic Analyze Telco and social data Implementation Identify the Implementation pattern(s) Develop a maturity Model Security/intelligence extension: Develop a Big capability maturity model Develop/use BDA Reference architecture Decision to utilize IBM Big Reference Architecture 53

54 Best Practices Security intelligence example deployment steps Identify deployment options BYO Hardware IT Appliances: Private Cloud: Deployment Identify security requirements infrastructure security needs; Security intelligence needs Risk management Monitor big data activity from applications and users for threats. Establish data traceability and auditability; Develop a BDA Governance Information is understood; Information is correct Information is holistic; Information is current Information is secure; Information is documented 54

55 Security intelligence example maturity model Best Practices Ad-hoc; inconsistent; some PoCs / trials; stand alone Big Analytics environment 2. Managed 1. Initial planned & managed project specific implementation; Some hadoop skills; some insights with EDW 3. Defined Quantitative defined objectives; Predictable outcomes; End-to-end integration with EDW; 4. Quantitatively Managed Big organizational governance; continuous updates inline with business objectives 5. Optimizing 55 1.Evaluate & Pilot IBM Streams for in motion requirements. 2. Revue applicable Big patterns 3 Document selected patterns and solution components 4. Identify SIEM requirements 5. Analyze cloud deployment needs 6. Start on BDA Governance 1. Develop SIEM QRadar architecture; identify SW components for security surveillance 2. Develop data models 3. Pilot - Hadoop for Big storage 4. Apply Cognos & SPSS for projects Test BI features to Private Cloud 6. Extend and apply BDA governance 1. Develop in motion system components. 2. Migrate Phase I security surveillance monitoring to private cloud 3. Access analytics through i2 ANB 4. Identify SIEM requirements 5. Access X-Force reports. 6. Extend and apply BDA governance 1. Extend real-time detection and prevention of threat 2. identify patterns 3. Pilot - Hadoop for Big storage 4. Identify SIEM requirements 5. Establish BDA Governance across projects

56 The BDA RA Reference Architecture material is based on TeamSD and applies to the en9re sales cycle Understand Client Define Client Requirements Design Solu0on Detail Design to Define BOM Best Prac0ces DevOps Business Drivers Use Cases System Context Architecture Decisions Best Prac9ces Describe the key business drivers for the project, the KPIs or CSFs, and how they align with Cloud compuang. What are the funcaonal requirements expected from the Cloud and who are the key actors. Expressed as Use Cases. The system context should define the boundary of the Cloud, and the integraaons with OSS / BSS systems Clearly documented decisions on key architectural points including the raaonale for the decision. Define the overall Ameline, phases, and key milestones that will shape the plan and overall delivery. Func9onal & Non- Func9onal Requirements Architecture Overview Opera9onal Model Dev Ops 56 Content provided by BDA RA Business Drivers Use Cases Requirements, Security Architecture Overview, Architecture Decisions Operational Model, DevOps Best Practices NFRs should be defined to cover the volumes, capacity, scale, availability, security, operaaonal and monitoring aspects of the Cloud Architecture overview diagram should define the high level components, their placement. Technology Brief Define the boundaries of the project, inclusions, exclusions, dependencies, and align phases with milestones in the roadmap. Design and consider the components of the soluaon both at a physical and logical level. Design and consider the components of the soluaon both at a physical and logical level.

57 Dev Ops DevOps for Big and Analytics Applications Plan and Measure Project and PorYolio Management 57 Requirements Management Enterprise Architecture Develop and Test ApplicaAon Lifecycle Management ConfiguraAon and Change Management End- to- end Traceability Quality Management and Service VirtualizaAon for TesAng Development Process Management Release and Deploy ConAnuous Delivery Source ConfiguraAon Management Release Management Monitor and Op9mize ConAnuous Monitoring Customer Feedback

58 Lifecycle Dev Ops 58 Cross Industry Standard Process for Mining (CRISP- DM) Business Understanding Understanding PreparaAon Modeling EvaluaAon Deployment Big Informa9on Integra9on and Governance Find and Understand Big Prepare Big for Usage Defend and Build Confidence in Veracity of Big Secure Big and Comply with Privacy RegulaAons Audit and Archive Big IBM InformaAon Governance Unified Process

59 DevOps and Lifecycle Dev Ops sources Real-time analytics Actionable insight Enhanced applications Machine and sensor data Cognitive Customer experience Image and video Enterprise content Transaction and application data Information ingestion and operational information + + Enterprise warehouse mart mart Decision management Predictive analytics and modeling New business models Financial performance Risk Social data Third-party data Exploration, landing and archive Analytic appliances Analytic appliances Reporting, analysis, content analytics Operations and fraud IT economics Information integration & governance Discovery and exploration SYSTEMS SECURITY STORAGE 59 DevOps for Big Lifecycle

60 Backup 60

61 Glossary of Terms Platform as a service (PaaS): It is a category of cloud computing services that provides a computing platform and a solution stack as a service. Basically it includes the hardware, Operating system, storage and network and the middleware, frameworks together a solution. Software as a service (SaaS): Application software owned, delivered and managed remotely by one or more providers. The provider delivers an application based on a single set of common code and data definitions that is consumed in a one-to-many model by all contracted customers at anytime. SaaS is purchased on a pay-for-use basis or as a subscription based on usage metrics. Proof of Concept (POC): a realization of a certain method or idea to demonstrate its feasibility or a demonstration in principle, whose purpose is to verify that some concept or theory has the potential of being used. Big Volume: Volume is the obvious Big trait. Aggregates of data that use to be measured in Petabytes are now measured in zettabytes or a billion terabytes. Big Variety: The Variety characteristic of Big is all about trying to capture all of the data that pertains to our decision-making process. Big Velocity: is the rate at which data arrives at the enterprise and is processed or well understood. Big Veracity: is the term that refers to the quality or trustworthiness of the data. Information Ingestion: Is the process to extract, transform the raw data from many sources (traditional sources) and load into a data repository (data warehouse or data mart) to make it available for analytics. It includes staging areas and specialized transformation engines that are performing enrichment and restructuring operations on the raw data. It also includes the data quality and standardization process to ensure that the data is conformed to corporate standards and well understood by every one that needs to use it. 61

62 Glossary of Terms Enterprise Warehouse (EDW): Consist of a single environment that contains subject areas consolidate across divisions or line of business. is highly normalized and provides application neutral base access. The data model maps the system of records. The system manages very complex and heavy workloads. It requires high available and fast recovery capabilities. Operational Stores (ODS): ODS is another type of implementation with time-sensitive operational data that needs to be accessed efficiently for both simple queries along with complex reporting to support tactical business initiatives. In the traditional architecture, the analytical sources are created based on specific business needs. As new business requirements arise, a new data process is needed to generate the data and make it available for consumers (users and/or applications). Predictive Analytics and Modeling: Predictive analytics is a discipline that leverage advanced analytical algorithms (Linear Regression, Decision Tree, etc) to process historical data and create models that can make predictions about future outcomes. Metadata: Metadata Management is the capture, versioning, approval, usage, and analysis of the different types of metadata found in an Information Management environment. Metadata Catalog: contains the semantic definitions for business and IT terms, data models, types, and repositories. It provides functionality to browse, discovery, and search of metadata assets. Master : Are the key business data elements that may include information about customers, products, employees, suppliers, vendors, etc. and shared as a single source of basic business data across systems, applications, and processes for an enterprise. Reference : Is the data that defines the standard data domain values used within an organization. Examples of Reference are: units of measure, country codes, corporate codes, conversion rates (currency, weight, temperature, etc.), calendar dates, etc Information Provisioning: Various provisioning mechanisms for locating, retrieving, transforming and aggregating information from all types of sources and repositories. Landing / Deep Zone : Area for raw data for querying, exploration, data transformations, and pseudo archival (aka, online queryable archive). The Area integrates and modernizes with traditional Integrated Warehouses, Discovery, MDM 360 & Content Management Information Provisioning: Various provisioning mechanisms for locating, retrieving, transforming and aggregating information from all types of sources and repositories. 62

63 Purpose This presentation, derived from the IBM Big & Analytics Reference Architecture document is meant to drive consistency across targeted technical sales channels in the roll-out of the IM & BA reference architectures aligned to the 5 big data use cases and the industry scenarios. The purpose of IBM Big & Analytics (BD&A) Reference Architecture is to inform and guide sales, technical sales, services and other professionals who are involved in selling IBM solutions or deploying them with clients. The Reference Architecture is intended to be used by a wide range of professionals selling IBM software and designing end-to-end big data & analytics client solutions. The reference architecture can also be reused by IBM clients and partners to gain further insight on how to benefit from IBM Software in big data & analytics projects (appropriate disclosure is required). The target audience also depends on the specific work product included in the reference architecture. The Architecture Overview, Business Directions. Requirements, Technology Brief are expected to be consumed by business leaders and architects. 63

64 Big Exploration Exploration User Experience Analytics Experience Content Analytics Miner Application Builder SPSS Modeler Cognos BI Integration & Governance Streams BigInsights Explorer Content Analytics Warehouse Connector Framework CM, RM, DM RDBMS Feeds Web 2.0 Web CRM, ERP File Systems 64

65 Enhanced 360 view of the Customer SOURCE SYSTEMS CRM Name: Address: J Robertson 35 West 15 th Address: Pittsburgh, PA Cognos BI Cognos Consumer Insight ERP Name: Address: Janet Robertson 35 West 15 th St. Address: Pittsburgh, PA Legacy Name: Address: Jan Robertson 36 West 15 th St. Address: Pittsburgh, PA InfoSphere Master Management InfoSphere Explorer 360 View of Party Identity First: Last: Address: City: State/Zip: Gender: Age: DOB: Janet Robertson 35 West 15 th St Pittsburgh PA / F 48 1/4/64 BigInsights Streams Warehouse Unified View of Party s Information 65

66 Operations Analysis Raw Logs and Machine InfoSphere Streams Capture Stream Real-time Monitoring Identify Anomaly SPSS Modeler Historical Reporting and Analysis Decision Management Raw InfoSphere BigInsights SPSS Modeler Predict and Classify Aggregate Results Cognos BI SPSS Modeler Predict and Score Warehouse Store Results Federated Navigation and Discovery 66

67 Warehouse Augmentation 1 Pre-Processing Hub 2 Query-able Archive 3 Exploratory Analysis SPSS Modeler Explorer BigInsights Landing zone for all data BigInsights Information Integration Explorer Find and view the data Combine with unstructured information BigInsights Streams Real-time processing Cognos BI Streams Offload analytics for microsecond latency Cognos BI SPSS Modeler Warehouse Warehouse Warehouse 67

68 Telco: Real Time Contextual Marketing Campaign CDRs, Top Ups, Balance Enquiry Usage, Location & History Monitoring Current State & Predictions Per Customer Real-time Event Detection EDW Streams Real-time Event Detection and Predictive Analytics Campaign monitoring Event-based Campaign 68

69 Banking: Real Time Credit Card Campaign Transactions, Location Usage, Location & History Monitoring Current State & Predictions Per Customer Real-time Event Detection EDW Streams Real-time Event Detection and Predictive Analytics Campaign monitoring Event-based Campaign 69