Information Server 11.3 Overview. Kevin D Silva Client Technical Professional, InfoSphere Information Server

Transcription

1 Information Server 11.3 Overview Kevin D Silva Client Technical Professional, InfoSphere Information Server

2 Governance Concerns for Big Customers Integrate & Link Big Big as a Source Big as a Target Transformations Movement Integrate w/existing Enterprise Lineage & Impact Analysis Metadata Integration w/analytics Realtime & Federation Protect Big Activity Monitoring Masking Encryption On-Demand / In-Place Protection In-Line Protection (w/etl etc.) Active Detection & Alerting Audit & Archive Big Queryable Archive Structured and Semi-Structured Optimized Connectors to existing Apps Hot-Restorable On-the-Fly Immutable and Secure Access Automated Legal Hold Capability for Freeze Cleanse and Validate Big Accuracy and Entity Matching with Social De-duplication and Standardization of Machine In-line Cleansing with Integration Trusted Dashboard and Reporting on Quality Master Big Big as a Supplier Big as a Consumer Links between Big and Trusted Golden Records Leverage Master in Big Analytics Entity Resolution at Extreme Scale Out Levels Probabilistic Entity Matching

3 I< Bd Ma t Ie n> f o S p h e r e I n f o r m a t i o n A n a l y z e r Thinking about the Information Supply Chain External Sources Transform & Cleanse & Monitor quality in-flight to identify issues in integration Monitor your trusted systems (MDM, WH, new packaged app.) to sustain required data quality level Integrate Master Management (Packaged App.) Warehouse Analyze BI Applications Packaged Applications Report status & progress to the business Monitor quality at the source to address issues where information originates Information Server Supply metrics to governance initiative 2

4 InfoSphere Information Governance Catalog - Lineage View end-to-end data lineage and impact analysis across data sources Heterogeneous data flow reporting One-click view of end-to-end upstream and downstream data flows Fast display of complex flows Advanced filters support defining scope of displayed properties Supports definition of custom data elements and data flows, including APIs for scheduled updates Business Lineage display available for non-technical audiences Links to Stewards and Glossary provide business context for graph items Accessible from desktop applications and directly from Cognos 3 IBM Confidential Information

5 Information Server v11.3 Agile Integration Wherever your information resides, integrate it quickly and flexibly Information Governance Catalog Business Driven Governance Make decisions with confidence using trusted data at the point of impact InfoSphere Information Sustainable Quality Ensure information accuracy and quickly adapt to strategic business changes Quality Server Integration

6 Information Governance Catalog The one-stop shop for business and technical metadata Information Governance Catalog Enterprise-class collaboration platform for creating, monitoring, and reviewing business-critical metadata InfoSphere Information Anywhere access that aligns business and IT so that any project can be supported by governed, trusted data Quality Server Integration

7 InfoSphere Information Governance Catalog - Overview A meaningful directory of governed information Simple search Hierarchical view and navigation Features Comprehensive catalog of information assets Governed business vocabulary Enables the creation of stewards & assignment of responsibilities for information domains Supports Collections of assets for team collaboration Links business terms and information governance rules to information assets and operational rules Integrates with other products for operational governance Displays end-to-end lineage across information assets Benefits Aligns the efforts of IT with the goals of the business Provides business context and governance to information technology assets Establishes responsibility and accountability throughout the information development lifecycle Accelerates information development Dramatically increases business confidence in information assets 6

8 Users want an easy way to find data I need sample data to build requirements for our next project. I need data in my landing zone for developing my next BigSQL application I have to build new models for marketing and don t have the data on my platform. Business Analyst BigInsights Developer Scientist 7

9 Adminstrators/Stewards should govern this data I want the data to be governed by enterprise policies and rules I need to ensure trusted sources and manage the quality and redundancy I need to ensure that the data is protected and secured Enterprise Steward Steward Security architect 8

10 InfoSphere Information Governance Catalog - Compliance Declare Information Governance Rules and track compliance Information Governance Policy 9 Published Blueprint with Asset Links Information Governance Policies & Rules Declare the intended behavior of information Leverage business terms for defining functional scope Communicate precise intent for how information must be managed throughout its lifecycle: Discovery Modeling Master Reference Quality Archiving Privacy Security Movement Transformation Availability Design and publish Blueprint specifications with links to metadata assets and operational rules Link Information Governance Rules to information assets and implementation components: Governs data sources Implemented By operational rules and routines View operational metrics from data quality and other tools

11 Information Governance Catalog Information Governance Catalog Convergence of two market-leading metadata capabilities: Business Glossary - business focused metadata for terms, policies, rules Metadata Workbench - IT focused metadata for technical assets, lineage, impact analysis Combined interfaces further accelerate business and IT collaboration. Streamlines workflows where users needed access to both styles of metadata IBM Corporation

12 Manage Information Governance Catalog subscriptions in the Subscription Manager Information Governance Catalog Engage users proactively when data integration, quality and governance activities discover items of interest Speed up responsiveness to data quality issues Increase organizational trust in information through increased level of collaboration IBM Corporation 1

13 Subscription Manager Business Driven Governance Information Governance Catalog Provides one consistent mechanism to notify key stakeholders of events relevant to them. Initial release will cover event types for both data quality and business policy and metadata objectives. Quality: collects events from IA, as well as from QS & DS (via exception stage) Business Policy & Metadata: to customize the definition, review and approval processes Also forms the basis of our integration with the new Stewardship Center (see next section) IBM Corporation 1

14 Subscription Manager - Details Information Governance Catalog Register Event Types. For each type, Subscription Manager can directly send notifications, or notify a registered provider. Registered providers, such as Business Process Manager, must have been previously configured. Additional filters can limit the type of action, or the Asset being modified IBM Corporation

15 Stewardship Center Information Governance Catalog A single collaborative environment for the data steward business team to manage data governance initiatives. Leverages prebuilt and customizable workflows via IBM BPM for data issue and governance process tasks. Provides specific data remediation tooling to act on and correct data quality issues IBM Corporation 1

16 Stewardship Center Workflow Monitoring The Stewardship Center provides OOTB Dashboards for monitoring both Team and Process Performance Information Governance Catalog IBM Corporation

17 Understand & Collaborate provides context and relevance to information in the data lake Information Governance Catalog Shop for Smart Hover Collect & Share Allows any user to find data sources incontext -- tables, files, cloud structures, and more and explore rich detail to find the most relevant data Instantly view summary information for any data asset and then quickly act either to collaborate, annotate or integrate Organize data assets into collections, add context, and share with other users who are working collaboratively on an information centric project New graphical lineage viewer that quickly composes and renders even the most complex enterprise flows from across 1000s of operational runs IBM Corporation

18 Shop for allowing any user to find relevant data Information Governance Catalog Use semantic search to instantly find relevant data sources and assets Augmented by integration of both business and technical metadata IBM Corporation

19 Smart Hover easily access summary info for any object and then act directly Information Governance Catalog Hovering over any asset provides context information without needing to change screens Use action buttons to drill into details, collect assets, users, integrate data and more IBM Corporation

20 Collect & Share organize data assets into collections, add context, and share Information Governance Catalog Promote crossteam collaboration through shopping cart like experience of collecting assets of interest Team members can add insightful notes as they curate info in the data lake IBM Corporation

21 Lineage supports graphical lineage at enterprise-class scale Information Governance Catalog Display of complex flows based on lineage flow architecture and graphing Advanced filters support defining scope of displayed properties Mobile device support, including iphone/ipad IBM Corporation

22 Shop for Act on integrated flow when IGC used with Click Information Governance Catalog After users find information that suites their business needs, they can quickly jump into Click to integrate that data into a target data store relational, Hadoop or Cloud IBM Corporation

23 Quality Ensure information accuracy and quickly adapt to strategic business changes Information Governance Catalog Profile of to get a quick understanding of what is in the data as opposed to what people may thing the data is. InfoSphere Information Improve the quality of information through standardizing, matching, enriching and validating. Quality Server Integration Track quality of data to ensure it meets business requirements and take action when it falls out of guidelines IBM Corporation

24 Profile and Manage Quality Rules with Information Analyzer Profile / Perform data quality assessment Define business rules to monitor data quality Establish stewards for governance of data quality Identify data quality issues early to reduce project risks Monitor quality metrics over time for compliance Manage and Remediate Exceptions with the Quality Exception Management Console Share Quality Measures and Metrics with the owners, stewards and stakeholders with the Information Governance Dashboard IBM Corporation

25 Analyse data content - Information Analyzer You cannot manage (data quality) if you cannot measure Aligning goal & outcome to business impact Completeness The existence/presence of actual data (i.e. not null, not spaces) Structural Integrity type checks (date, numeric) format validation (by character or regular expression) Validity equality or difference (=, not =, >, <, >=, <=) string containment (e.g. contains TEST a search for strings containing this specific word) within a valid range Reference lookup (by list or to a reference table) Uniqueness (or frequency of occurrence, usually of a key or identifying field) Referential Integrity A key to one record exists in a corresponding list of keys in another table Valid-Value Combinations Multiple validity conditions together with AND or OR clauses (e.g. CustomerType = G AND CustomerStatus = A ) Computational rules Arithmetic expressions (fielda + fieldb = fieldc) or Aggregations (e.g. Sum, Count) Time rules Ordered sequencing of dates If Then Else rules Conditional expressions that may combine any of the above constraints, but only in relation to a segment or subset of the data (e.g. only IF CustomerStatus = A )

26 Applying Information Analyzer Rules Specify consistent & re-usable data rules, driven by business The account number must meet the following condition: driven by Rule Business users validated against Examples of Rules: The Gender field must be populated and must be in the list of accepted values The Social Security Number must be numeric and in the format If Date of Birth Exists AND Date of Birth > and < TODAY Then Customer Type Equals P The Bank Account Branch ID is valid in the Branch Reference master list IBM Corporation

27 Quality Exception Management Updates Quality User refresh includes enhanced search filter to drill into exception set information Now, selecting an exception set allows the user to it manage with the new Stewardship Center features Actions in the Stewardship Center will also be reflected her as the current status, owner and last updated are modified IBM Corporation

28 Exception SQL Views SQL interface to content of metadata repository Quality Support creation of custom reporting solutions Abstract from specifics of the metadata schema Hide database or application specific implementation details Represent key metadata in consumable format Component-specific schemas: Common Metadata (cmviews) Quality Metadata (iaviews) Glossary Metadata (igviews) Exception Management (remviews) Documented in IIS Knowledge Center: Information Governance Dashboard > Implementing your own information governance reports > SQL views IBM Corporation

29 Stewardship Center - Remediation Quality Provides details of exceptions captured during data integration, data cleansing and data profiling activities. Drill into rule details in IGC to understand how this exception set relates to business objectives. Edit values that were incorrect or use the Export / Import buttons to download / upload a CSV file with the records and associated fixes. Prebuilt workflow includes review and approval steps for multiple organizational roles ( Steward, Steward Manager, Source Owner) Prebuilt workflow can be extended/customized to allow approved edits to be submitted to any REST service for automated cleanup IBM Corporation

30 Information Governance Dashboard visualize how data quality objectives are aligning to policies Quality IGD is the Integrated Quality Reporting capability Over 100 reports including analysis of critical business elements that lack a data steward owner Cognos now entitlement to support rollout of data quality program IBM Corporation

31 Operational Quality measure how platform is meeting expectations Quality Pre-built rules to measure 4 domains of operational quality Pinpoints opportunities to optimize the environment Provides unified environment to assess and monitor data for data stewards, analysts, etc IBM Corporation

32 Information Quality Process Goals: what is the end state? Scope: what is included/excluded in the effort? Identify & Document Requirements: what are the key requirements? Quality what are the business requirements and terminology? IBM Corporation

33 Integration Transform and deliver any data at any level of complexity or scale Information Governance Catalog Self-service data integration so that any user can connect to and move data on-demand InfoSphere Information Market-leading massively scalable data integration engine to refine data so that it is fit for business purpose Quality Server Integration IBM Corporation

34 Transform & Deliver refining data at any scale to make it fit for business purpose Integration Self-service Cloud Reach Master Sort Compress Allows any user to get or share data in or out of the data lake through a simple browser based UI leveraging the scalable DI engine & rich metadata Helps enterprises who are shifting operations and analytics into the cloud, to provide data to these environments and to collocate the DI platform with them Speed-up time to integrate data into your MDM solution through MDM model exploration and simple dragand-drop integration from and to any other data source For large data sets, reduces I/O size for temp files thus improving run time - benefits both explicit sorts and those that are inserted in specific operators IBM Corporation

35 Click self-service data integration for the data lake Integration Allows any authorized user to get/share data Automatically registers metadata Enhancements include New home screen Monitoring Web-based authoring IGC integration Expanded connectivity (Amazon S3, Hadoop, ODBC, JDBC, native DB2 and Oracle) IBM Corporation

36 Cloud Reach extending the enterprise s reach off-premise Integration Easily move data into or out of your cloud hosted Amazon S3 environment Hierarchical stage adds REST APIs to interact with on & off-premise sources, including DBaaS offerings such as IBM Cloudant IBM Corporation

37 InfoSphere Click Integration Self-service data integration on demand Allow any user to get data quickly data scientists, business analysts, report developers Only a few simple clicks move data from a source to a target load a sandbox, put data in a staging area, create a mart Works with a variety of heterogeneous data store technologies Pure (DB2, Netezza), Teradata, Oracle, SQL Server, Greenplum, Hadoop, and many more..

38 Click: simple access to move data Integration Review data movement activities that are already available or build new ones Examine history of what had been run, its status and how much data was moved Select one or more data sources that need to be moved into BigInsights through a few simple steps.

39 Click: Supported data source combinations Sources DB2 Targets HDFS Netezza Teradata Oracle SQL Server Informix Greenplum Sybase MySQL Progress OpenEdge PostgreSQL GPFS Hive Amazon S3 Elastic Storage All of the above technologies listed in Sources

40 Comprehensive features for data integration Integration Quality Connectivity Parallel Engine Dozens of prebuilt transformation objects & 100s of functions for even the most complex requirements Standardize, match, cleanse, enrich and validate data across multiple information domains all in same design tool with DI bases, Big, Messages, CDC, Mainframe, Streaming data, files, XML, industry formats, cloud, ERP and extensibility Most scalable data integration engine supporting SMP, MPP and Grid with simple configuration file flexibility One Design Environment Single UI for design, development, compile, execute, test/debug and performance analysis to accelerate learning curve and time to value One Set of Design Artefacts Logic represented by one set of design objects regardless of deployment styles One Metadata Store Maximizes business & IT collaboration and accelerates data governance efforts and provides advanced impact analysis, search, comparison and more. One Administration Center Integration of install, security, auditing, connectivity, logging reduces TCO. Multiple processing styles Traditional scheduled batch or SOA real-time using web services, RSS, REST, JMS... Distributed Transactions Scalable, heterogenous information fabric with guarenteed data delivery (two phase commit) Balanced Optimization Maximize infrastructure by moving processing to DBMS or Hadoop (ELT) Workload Management allows policy based assignment of resources and priorities for jobs to maximize system utilization Enterprise Packs Specific connectivity for leading ERP solutions Self Service Web-based interface lets business users access and deliver data to achieve agility in as little as 2 clicks. Business Rules Drive critical enterprise logic through OOTB integration w/ market leading Jrules Masking Protect sensitive information such as personally identifiable information Operations Console provides intuitive graphical display of job and system information to support analysis of the runtime environment 39

41 Thank You ibm.com