Big Data Platform Implementation

Size: px

Start display at page:

Download "Big Data Platform Implementation"

Hugh Pierce
5 years ago
Views:

1 Big Data Platform Implementation Consolidate Automate Predict Innovation Intelligence Cloud

2 Big Data Platform Implementation - Objective InnoTx helps organizations create an Analytics Ready Data environment. This helps in creating a single, unified data management framework that brings appropriate data together for reporting & analysis. Such streamlining of data provides a unified view across various segments. Manage huge volumes of data, with multiple deployment options. Automation of data pull, aggregation for reporting and increase in productivity Gain deep insights to take timely action

3 Big Data and Analytics Platform

4 Big Data Platform - Benefits All data has potential value No defined schema - stored in native format Schema is imposed and transformations are done at query time (schema-on-read) ELT(Extract, Load, Transform) rather than ETL processing Designed for low cost storage Highly agile configure and reconfigure as required Provide Faster Insights Apps and users interpret the data as they see fit ELT WORKFLOW

5 Big Data Platform Data Lake Ecosystem DESIGN IMPLEMENT ANALYTICS SUPPORT

6 Big Data Platform Technical Architecture Full Stack Support

Big Data Platform Technical Architecture PHASE 1: Data Set-up and Movement

Analyzing the architecture of databases, studying table structures, etc.

Installation of Big Data platform on respective project server ii.

Data movement i. Identifying data to be migrated ii.

Migrating data to project server Activities include: Hardware Setup and

Testing Testing (including sample data) Data Transfer- Final Acceptance

7 Big Data Platform Technical Architecture PHASE 1: Data Set-up and Movement Analysis of data sources i. Identification of data sources ii. Analyzing the architecture of databases, studying table structures, etc. iii. Data sizing for planning hardware requirements Technology setup i. Installation of Big Data platform on respective project server ii. Installation of supporting applications such as MS office, etc. Data movement i. Identifying data to be migrated ii. Creation of input databases and output databases iii. Migrating data to project server Activities include: Hardware Setup and Testing Connectivity Big Data Platform Download and Installation Coherency Testing Testing (including sample data) Data Transfer- Final Acceptance Testing activity Testing and validation Testing data lake architecture, UAT, validation reports, etc. *Each process involves InnoTx team and Client support

Big Data Platform Technical Architecture PHASE 2: Transfer and Loading Data for End Use Creation of assets data dictionary Data Model creation i.

Mapping of input tables/variables to output tables iii. Scripts for output table creation iv. Data mart creation Generation of reports i.

Run queries and output tables in consumable form Activities include: Understanding requirements for Metadata and final Use Cases Document Metadata implementation

8 Big Data Platform Technical Architecture PHASE 2: Transfer and Loading Data for End Use Creation of assets data dictionary Data Model creation i. Selection of relevant input tables ii. Marking relevance of variables in each table i. Specification of output table ii. Mapping of input tables/variables to output tables iii. Scripts for output table creation iv. Data mart creation Generation of reports i. Understand the specific report requirements ii. Reuse the existing queries to create Hadoop queries iii. Run queries and output tables in consumable form Activities include: Understanding requirements for Metadata and final Use Cases Document Metadata implementation Design/Implementation Testing of final structured data Regulatory Reports generation & validation Acceptance Testing BI tool integration i. Establish connection between BI tool and Data Mart ii. Generation of intermediate data cubes for visualization iii. Demonstrate generation of reports via the BI tool *Each process involves InnoTx team and Client support

9 Delivery Process User review 1 Business requirement gathering and sign offs 2 Specific data sets identification as per mutual discussion 3 Data sizing requirements 4 Technology Setup 5 Validation to establish data integrity Analysis of data to track industry trends from the data Reporting Insights, dashboards/r eports and output analysis

10 Delivery - Success Metrics Deliverables/Milestones Infrastructure Setup Specification Documents Assets Data Dictionary Data Movement Pipeline (Data Pipe) Archival Data Store Reports Documentation Assets Data Mart Assets Reports Query Performance Assessment Real-time Dashboards via Spotfire Success Criteria Infrastructure verified as per information provided in the proposal and subsequent s Excel with Tables, Variable names and their importance etc Scripts that enable automated movement of data; single click run and query tests for newly moved data Basic stats comparison, eg.. row counts, sum of columns, unique entries etc Business Requirements Document: Report output values, input variables, frequency, updates, and usage (department wise) Reports enabled with quick query respose wrt raw data Tables and Graphs as per BRD Excel Tables and Graphs for execution time for Data ingest, Compression store, Data Cube building, Report generation etc Spotfire real-time interactivity with underlying Hadoop data

11 Implementation Team Client Engagement Manager Business Analyst Requirement specification Data Scientist Data & Platform architecture, Advisory & Sizing Data Modeler/Analyst - Design architecture and Data Model Big Data Engineer Platform Implementation and ETL Data Analyst/s : Analytical work load Specialist Support if any, with vendor Involvement (If any, like Cloudera support) Post Go Live Admin Roles for Maintenance & Risk Assumption Delivery is on onsite basis

Big Data Platform Implementation - Impact Case Study1: Big Data Implementation in payments & transactional intelligence Client Context : Nodal agency for payments serving 600+ Banks/Payment product

12 Big Data Platform Implementation - Impact Case Study1: Big Data Implementation in payments & transactional intelligence Client Context : Nodal agency for payments serving 600+ Banks/Payment product providers Benefits: Implementation costs 1/10 th of similar implementations Automated reporting, Complex query for analysis (for 6 products, multiple stakeholders across Banking/Govt) High volume, High velocity, Real time, User Behaviour use cases

Big Data Platform Implementation - Impact Case Study2: Leading Private Sector Bank Client Context : Implementation of a Big Data platform (Hadoop) on premise on existing virtualized IT infrastructure

13 Big Data Platform Implementation - Impact Case Study2: Leading Private Sector Bank Client Context : Implementation of a Big Data platform (Hadoop) on premise on existing virtualized IT infrastructure Building a fully functional Assets Data Marts using data model, corresponding regulatory reports, and interactive visualization Implementation and testing of various data security and system security processes and technologies required for Banking applications and use-cases Benefits: Archival data storage Performance DBs Real-time integration Secure environment

14 Customers Analytics Services Insurance Analytics Suite

15 Your Digital Transformation Partner A Snapshot

Innovation Aggregators Leverage digital technologies to

transform business processes based on Innovation Intelligence

It is central to digital transformation.

16 Innovation Aggregators Leverage digital technologies to disrupt business models, re-design customer experiences and transform business processes based on Innovation Intelligence Cloud Innovation is no longer an option. It is central to digital transformation. There is no dearth of data. Intelligence will be the differentiator Innovation at scale is not possible without Cloud.

We bring disruptive technologies to create a first mover environment provide a competitive edge provide a platform to innovate Industry

17 Digital Transformation Stack InnoTx is your digital transformation partner. We help you translate your digital vision, and transform your business into a digital business. We bring disruptive technologies to create a first mover environment provide a competitive edge provide a platform to innovate Industry Transformation Government Utilities & Energy Retail Banking & Insurance Healthcare & Hospitality Education Innovation Intelligence Cloud Technology-led Disruption AI or Cognitive IoT einsurance RPA Big Data Analytics Resourcing Private, Public, Orchestration

offering Advisory Big Data Analytics Software Development Cloud

enable CIO/CTO/CDO in developing services portfolio for digital

We challenge your vision and accelerate your digital Quantum vs

18 Who we are. What we believe in Middle East arm of Teckraft Infosolutions Group, estd Team of 180+ professionals spread across India, MENA and Europe offering Advisory Big Data Analytics Software Development Cloud Resourcing Solutions MENA office based out of Abu Dhabi, UAE We enable CIO/CTO/CDO in developing services portfolio for digital transformation. We challenge your vision and accelerate your digital transformation. Quantum vs Incremental changes Making the Innovative and disruptive technology accessible locally. We enable experimentation. PoCs are essential Rooted and aligned to Govt s vision on Innovation and backed by UAE national s investment

19 Clients Select List Logistics Pharmaceuticals ITES BFSI Manufacturing & Other International 19

20 Thank You PO Box Abu Dhabi UAE

Big Data Services Portfolio. Analyse Predict Monetise

Big Data Services Portfolio. Analyse Predict Monetise Big Data Services Portfolio Analyse Predict Monetise Services Portfolio BIG DATA ADVISORY BIG DATA PLATFORM IMPLEMENTATION CALL CENTER OPTIMISATION RESOURCE AUGMENTATION Services Big Data Advisory Consolidate