Collibra Catalog for Big Data Analytics Product Preview

Size: px

Start display at page:

Download "Collibra Catalog for Big Data Analytics Product Preview"

Evelyn Rogers
6 years ago
Views:

2 Collibra Catalog for Big Data Analytics Product Preview Stan Christiaens Co-founder & Chief Technology Officer, Collibra May 3, 2016

3 Disclaimer The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Collibra products remain at the sole discretion of Collibra. 3

4 Today s Agenda State of the Market Demonstration: Collibra Catalog Workflows & Timeline Preparing for Collibra Catalog 4

5 The BI Market is in a Serious State of Transition A continued focus on data and analytics, but with mixed success 86% of executives say their organizations have been, at best, only somewhat effective at meeting their primary objectives More than 25% report being ineffective A massive shift towards self-service BI But through 2016, less than 10% of self-service BI initiatives will be governed sufficiently to prevent inconsistencies that adversely affect the business Sources: The Need to Lead in Data and Analytics, McKinsey, April 2016; Best Practices for Driving Successful Analytics Governance, Gartner, August

Top (Analytics) Talent is Hard to Find 48% of companies find it more difficult to attract analytics talent 42% of companies find it more difficult to retain analytics talent

6 Top (Analytics) Talent is Hard to Find 48% of companies find it more difficult to attract analytics talent 42% of companies find it more difficult to retain analytics talent Business users with analytics skill sets are the hardest to find, followed by data scientists or engineers Source: The Need to Lead in Data and Analytics, McKinsey, April

Data Scientists Can t Do Their Jobs Accessing data is difficult and timeconsuming Time to impact is slow Identifying ownership across on-premises, cloud, and external

7 Data Scientists Can t Do Their Jobs Accessing data is difficult and timeconsuming Time to impact is slow Identifying ownership across on-premises, cloud, and external solutions is challenging Fostering collaboration and reuse is challenging Multiple data sources are in use; some are outdated Many variants and versions of the same data exist 7

8 They Spend Most of Their Time on the Least Enjoyable Tasks Data scientists spend 60% of their time cleaning and organizing data But 57% of data scientists view this as the least enjoyable part of their job Source: CrowdFlower via Forbes 8

9 Enabling Analytics 1. Identify and certify trusted sources 2. Formalize responsibilities 3. Establish data quality metrics 4. Watermark outputs 5. Make lineage visible at point of consumption (context via metadata) 9

queries, import jobs Figure out the joins, mash it up Iterate quality/data prep Figure out the business logic Clean the data Build dashboards/ mobile apps Validate it with stakeholders (IT, CXO,

10 Typical Analyst Tasks Analysis Tasks Current Approach Time Required Find the data Ask around, wait for answer. How to connect to a data steward? Search internally Power users have rights to go get external data Understand the data Schema checks Data sample, find related data Figure out business definition & rules Import the data you want Run queries, import jobs Figure out the joins, mash it up Iterate quality/data prep Figure out the business logic Clean the data Build dashboards/ mobile apps Validate it with stakeholders (IT, CXO, inter-department) Presentation layer Discover new patterns. Make sure data is valid, quality controls, compliance, policies are right: stakeholders. Back & forth process Risk of BI project failing 4-6 weeks 2-3 weeks Iterative process Months The iterative process of data discovery is directly dependent on how comfortable the analysts are with finding the right data, understanding it, and making sure all stakeholders are onboard with the analysis. 10

- Search Internally - Power users have rights to go get external data Understanding the data - Schema Checks - Data Sample, find related data

Quality/Data prep - Figure out the business logic - Clean the data Build Dashboards/Mobile Apps - Presentation Layer - Discover new patterns.

$Back & forth process - Risk of BI project failing Collibra Data Governance Center 4-6 Weeks 2-3 Weeks Data prep\data quality Months Iterative$

11 Typical Analyst Tasks Analysis Tasks Current Approach Time Required Find the data - Ask around, wait for answer, How to connect to a Data Steward? - Search Internally - Power users have rights to go get external data Understanding the data - Schema Checks - Data Sample, find related data - Figure out Business Definition & rules Import the datayou want - Run Queries, Import Jobs - Figure out the joins, Mash it up Iterate Quality/Data prep - Figure out the business logic - Clean the data Build Dashboards/Mobile Apps - Presentation Layer - Discover new patterns. Validateit with Stakeholders (IT, CXO, inter-department) Analytics tools Collibra Catalog - Make sure data is valid, quality controls, compliance, policies are right: Stakeholders. Back & forth process - Risk of BI project failing Collibra Data Governance Center 4-6 Weeks 2-3 Weeks Data prep\data quality Months Iterative process Iterative process The iterative process of data discovery is directly dependent on how comfortable the analysts are with finding the right data, understanding it, and making sure all stakeholders are onboard with the analysis. 11

12 Demonstration

13 Collibra Catalog in Detail

$data Provides a process around building\augmenting the metadata repository of all data assets Delivers a$

14 Cataloging a Data Lake Swamp Google for metadata Provides an overview of how data flows in an enterprise Captures metadata that provides structural, semantic, lineage, and usage information Amazonification of data Provides a process around building\augmenting the metadata repository of all data assets Delivers a suggestion/inference engine that learns user intentions and recommends data sources and related objects 14

15 Applying Data Governance Data Governance and Stewardship Capabilities Added by Collibra Data Governance Center Contextualizing Data Establishing Ownership Data Quality Insight Data Stewardship business glossary data dictionary critical data elements reference data staging mappings and crosswalks metrics catalogue policy management voting and approval escalation and notification process traceability business rules metrics and monitoring DQ dashboards issue management lineage data sharing agreement data ingestion requests promotion and context certification and watermarking Collibra Data Governance Center Foundation Dashboard Organization People Roles Process Information Assets API Your existing data operations data consolidation & archival data sources / applications / external data data movement & data cleaning ODS MDM DWH Data Mart virtualization & analytics Big Data Discovery Big Data Archival 15

16 Applying Data Governance Data Governance and Stewardship Capabilities added by Collibra Data Governance Center Data Contextualizing Data Establishing Ownership Data Quality Insight Data Stewardship Business business glossary Reference metrics catalogue Data Stewardship business / rules Policy Manager data sharing agreement data dictionary Glossary policy management metrics and monitoring data ingestion requests critical data elements voting and approval Data Helpdesk DQ dashboards promotion and context reference data staging escalation and notification issue management certification and mappings and crosswalks process traceability lineage watermarking Data Catalog Collibra Data Governance Center Foundation Data Governance Platform Dashboard Organization People Roles Process Information Assets API Your existing data operations data sources / applications / external data data movement & data cleaning Collibra Connect data consolidation & archival ODS MDM DWH Data Mart virtualization & analytics Big Data Discovery Big Data Archival 16

17 Benefits of Collibra Catalog Collaboration Fosters engagement by providing context on the data for use across the organization Enables quick identification the data owner to easily resolve questions and issues Governance controls Takes advantage of existing roles, workflows, and dashboards Leverages existing operating models Content Uses existing data elements and policies 17

18 Collibra Catalog: Primary Workflows Data ingestion Data provisioning 18

19 Collibra Catalog: Data Ingestion Tell Catalog you want to ingest a data source Create data source 19

20 Collibra Catalog: Data Ingestion Tell Catalog the type of data source you are going to ingest Create data source 20

21 Collibra Catalog: Data Provisioning Request access to data set On the data set page, you can request access 21

22 Collibra Catalog: Data Provisioning Request access to data set This will launch the request access dialog and will create a data usage asset 22

23 Collibra Catalog: Timeline and Beta Program Release Dates Enterprise Catalog available Fall 2016 (Collibra 5.0) After Enterprise Catalog adding support for Self Service Catalog Beta Program: Pre-Prototype Conduct usability testing Prioritize features and understand customer requirements Refine requirements based on customer feedback Beta Program: Post-Prototype Conduct exploratory testing and enablement Co-create new requirements and minor improvements 23

Contact your account representative Schedule an

Preparing Now Identify the number of new users and

Manager for a Day Session Tomorrow at 7:30am advance

24 Contact your account representative Schedule an in-depth demo Discuss pricing Plan internally Start Preparing Now Identify the number of new users and confirm sizing Budget accordingly Attend the Product Manager for a Day Session Tomorrow at 7:30am advance sign-up required! Join the Beta Program Sign up at 24

25 Questions?

26 Thank You Tell us how we did citizens.collibra.com/feedback

In search of the Holy Grail?

In search of the Holy Grail? Our Clients Journey to the Data Lake André De Locht Sr Business Consultant Data Lake, Information Integration and Governance $ andre.de.locht@be.ibm.com ( +32 476 870 354 Data