InfoArchive: Ensuring Big Data Compliance and Reducing Risk with Real Time Analytics

Size: px

Start display at page:

Download "InfoArchive: Ensuring Big Data Compliance and Reducing Risk with Real Time Analytics"

Giles McDonald
6 years ago
Views:

2 JOIN THE CONVERSATION! #MMTM16 ECD SERVICES Take the LEAP personality quiz and win! Genius Labs Garden Level Foyer Connect with us 2

3 An Enterprise Information Archiving Platform that unlocks data of all types, trapped in siloed applications, lowering IT costs, preserving compliance and putting application data to work. Leave No Application Data Behind 3

4 4

5 Conflict: Analytics vs Compliance ANALYTICS Access to as much data as possible? Protect & control access to data COMPLIANCE Access = ƒ(compliance) Without proper controls, the compliance, risk, and/or legal teams will block efforts to move data into the lake. 5

6 Analytics Use Cases HISTORICAL Make better decisions based on understanding what happened REAL TIME Make better decisions based on what is happening 6

7 Historical Analytics Data Source Ingestion Archive Access Data Source ediscovery Data Source Real Time Analytics Compliance Engine Historical Analytics Act WORM Storage Act 7

8 Compliance and Data Lake Compliance Data Filter compliance data into the Data Lake Application data streaming into data lake DATA LAKE Build compliance into the Data Lake? 8

9 Bringing Compliance to Analytics Customer needs compliant archive for application decommissioning, active archiving & data reuse Customer needs scalable big data analytics Application built using Hadoop & 3 rd party tools Hadoop InfoArchive HDFS MapReduce Big Data Analytics Big Data Analytics 9

10 InfoArchive and Hadoop Real time access to the information InfoArchive REST/JDBC Option 1: Direct Access Hadoop Retention policy Filter Transform Tokenize Mask Information proactively XML XML XML pushed HDFS API XML XML XML Option 2: Rendition Hadoop 10

11 InfoArchive and Hadoop Analytical Rendition Analytical Rendition Partial masking and redaction Transformation to suitable format, e.g. JSON Retention policy is inherited from the source data <?xml version="1.0" encoding="utf-8"?> <Customers> <Customer CustomerID="GREAL"> <CompanyName>Great Lakes Food Market</CompanyName> <ContactName>Howard Snyder</ContactName> <ContactTitle>Marketing Manager</ContactTitle> <Phone>(503) </Phone> <SSNBR> </SSNBR> <FullAddress> <Address>2732 Baker Blvd.</Address> <City>Eugene</City> <Region>OR</Region> <PostalCode>97403</PostalCode> <Country>USA</Country> </FullAddress> </Customer> <Customer CustomerID="HUNGC"> <CompanyName>Hungry Coyote Import Store</CompanyName> <ContactName>Yoshi Latimer</ContactName> <ContactTitle>Sales Representative</ContactTitle> <Phone>(503) </Phone> <SSNBR>XXX-XX-XXXX</SSNBR> <FullAddress> <Address>City Center Plaza 516 Main St.</Address> <City>Elgin</City> <Region>OR</Region> <PostalCode>97827</PostalCode> <Country>USA</Country> </FullAddress> </Customer> <Customers> Mask Delete 11

12 Customer Use Cases FSI Understanding customer interactions Compliance & Risk Healthcare Understand treatment results to predict new approches Predicting readmission rates 12

13 Real Time Analytics Data Source Ingestion Archive Access Data Source ediscovery Data Source Real Time Analytics Compliance Engine Historical Analytics Act WORM Storage Act 13

Real Time Use Cases DATA CENTRIC INGESTION CENTRIC

Collection Deduplication Automatic classification

Management Update Elimination Scale-out ingestion

delivery Digital Surveillance Fraud Prevention Data

Dodd-Frank, MiFiD II Right to be Forgotten Right to

14 Real Time Use Cases DATA CENTRIC INGESTION CENTRIC DOMAIN CENTRIC COMPLIANCE CENTRIC Metadata Collection Deduplication Automatic classification Data partitioning strategies Entity Extraction Index Management Update Elimination Scale-out ingestion processing Fault-tolerant processing Guaranteed data delivery Digital Surveillance Fraud Prevention Data Loss Prevention Data Valuation Sentiment Analysis Dodd-Frank, MiFiD II Right to be Forgotten Right to Access Retention Decisions Encryption Decisions Tokenization Decisions 14

15 High Volume Ingestion Framework Ingestion Framework Microservices InfoArchive Spring Cloud Data Flow Cloud Foundry, Yarn, Mesos, Kubernetes 15

16 Adaptable Ingestion Framework Microservices InfoArchive 16

17 Real Example Ingestion Framework Microservices Extract Enrich Archive (dedup and package) InfoArchive Confirm LDAP 17

18 Real Example Continued Ingestion Framework Microservices Extract Enrich Sentiment Archive (dedup and package) InfoArchive Confirm CoreNLP LDAP 18

19 The Big Picture Text Analytics Real-Time Processing Advanced Text Processing Machine Learning Natural Language Processing Many Frameworks Similarity Calculations Adaptive Learning Entity Recognition Microservices Vectorization Classification Sentiment Analysis... Shallow Learning Topic Modeling Language Dependencies 19

20 Conclusion & Call to Action InfoArchive enables both historical and real time analytics while remaining compliant Historical Analytics Direct or via rendition Real Time Analytics When data is ingested Enrich or take action Feedback: What are your archiving use cases? What are your analytics use cases? 20

21 MOMENTUM BARCELONA APP AND WIN! BEYOND SILOS Play the BEYOND Game and win a Raspberry Pi pre-loaded with InfoArchive 21

22 LET US KNOW WHAT YOU THOUGHT Take the Session Survey 1. Open the schedule with the Momentum App 2. Go to the session you attended 3. Open Session Survey 4. Answer the 4 questions and submit. Thank you! 22

23 23

MOMENTUM BARCELONA APP AND WIN!

MOMENTUM BARCELONA APP AND WIN! BEYOND SILOS Play the BEYOND Game and win a Raspberry Pi pre-loaded with InfoArchive http://bit.ly/mmtm16bcn 1 InfoArchive: A Customer Journey Alan Hutcheson Tord Svensson