Essential Elements and Metrics for a Data Warehouse TCOE. Amita Awasthi Infosys Limited (NASDAQ: INFY)

Transcription

1 Essential Elements and Metrics for a Data Warehouse TCOE Amita Awasthi Infosys Limited (NASDAQ: INFY)

2 Abstract We know from our experience that Data warehouse is a must for large organizations, as it provides insight into huge volume of data and enables them to take business decisions. Testing of data warehouse becomes a critical factor as any issue with the quality of data in the data warehouse can lead to huge issues. It is not only a functional testing area but also a topic of research where we see rapid evolution of tools and technology, the latest trend is Big Data which can support all the 3 Vs(Volume, Velocity, Variety) of data which are big challenges in Data Warehouse. All the top Data warehouse appliances, ETL, BI tool vendors are in the race to extend their offerings to support hadoop and other Big Data platforms. Big data may also become a source of information to our regular Enterprise Data Warehouse where it can feed in the unstructured data and helps in advance analytics. Researchers are working to see how maximum benefit can be achieved by combing EDW and Big Data. There are other advancements as well like Data warehouse on cloud, Mobile Business Intelligence etc. For any organization to keep a tab of these advancements and extract maximum benefits out of the data it is very much required to have a dedicated Data Warehouse Testing Center of excellence in place. This paper is to elaborate and discuss the essential elements and metrics for a data warehouse testing center of excellence 2

3 Abstract The information provided in this paper is a result of work done in defining the data warehouse testing center of excellence roadmap for 2 clients in last 8 months. KM Templates Quality KPIs Staffing Process Methodologies Automation ROI DWT Project 1 Estimates Test Strategy Project Mgmt. DWT Project 2 FSI Client DWT Manager Infra/Tools Test Environment Management Tools Management DWT Project 3 People Project Manager Technology SME Solution Architect DWT Project 4 Tools SME Domain SME DWT Project 5 Group 1 Group 2 Group 3 Group 4 Key Challenges Lack of DWT skilled resources, no competency development plan in place Spending too much time in collecting data/metrics. DWT specific metrics not defined Not able to focus on the latest trends in DWT space in market No centralize repository for processes, training, tools, SMEs, Best practices, templates etc. No clear direction on career opportunities for DWH tester Lessons learnt, best practices from similar project is not documented and shared across all DWT project This paper explains the essential components and benefits of moving to a DWT COE. People Process DWT COE Technology 3

4 Key Takeaways Characteristics of a Data Warehouse TCOE Metrics specific to Data Warehouse Testing Data Warehouse Competency enablement framework Latest technology trends in Data Warehouse and how QA is prepared for them How your existing testing landscape can be transformed to Data Warehouse TCOE 4

5 Target Audience Audience Prerequisite- Basic Knowledge of Databases, Data warehouse testing Intended Audience- Managers, Technical Leads and Testers of a Data Warehouse Testing Project 5

6 Speakers Profile Amita Awasthi is a PMP certified Project Manager with Infosys. She did her B Tech from HBTI, Kanpur. During her 13 years at Infosys she has gained experience in handling large virtual teams and different type of clients, projects, people and technologies. She has been recognized at organization level and a winner of Infosys Excellence Award, KM Trailblazer Champion and People s Manager. She is a SME for Data warehouse testing and Infosys DWH testing solution called Perfaware. Thought leadership, knowledge management, Project & Program Management, Data warehouse testing and Big Data are her key areas of interest and she has presented papers in Internal and external forums ( Currently she is managing multiple projects for a major US based Banking Customer, and actively contributes to unit level activities. The author can be reached at amita_awasthi@infosys.com 6

7 Essential Elements and Metrics for a Data Warehouse TCOE 7

8 Context & Background In today s world we all rely on data and make informed decisions, for any large organization data warehouse is the holy grail of information which is helping them to analyze the past, make decisions for today and future. It is no more limited to after the fact analysis with the advent of continuous technology innovations in this space. Managing and executing the data warehouse testing projects has become more challenging and interesting as the service offering itself is getting refined with the latest technology trend. We have seen that many of our clients are struggling with the decentralized way of managing DWH testing projects and either moved or have future roadmaps defined to move into the DWH testing Center of excellence model. According to Gartner Hype Cycle for Information Infrastructure, 2012, the Logical Data Warehouse (LDW) is a new data management architecture for analytics which combines the strengths of traditional repository warehouses with alternative data management and access strategy. The LDW will form a new best practices by the end of There are some essential elements, metrics and roadmap definition for transforming to Data Warehouse TCOE Reference: 8

9 Context & Background The objective of this paper is to elaborate on the three essential elements of Data warehouse TCOE 1. People 2. Process 3. Technology The solution provided here talks about the challenges faced by clients in a traditional data warehouse testing set up, what is the market perspective and trend that we are seeing in current times. This can be used as a skeleton framework to access the current DWH testing state, and outlining the roadmap for moving to a mature DWT COE end state. There are metrics defined specific to data warehousing which are crucial for data quality and load. 9

10 DWH Testing Challenges in a typical implementation Data Publishing Source Staging Data Warehouse Reporting and Analytics Data Marts ETL ETL Summary data Raw data Reports & Dashboards Ad hoc analysis Outbound Extracts Mobile Apps Metadata In-memory databases Data Quality checks not performed on source system data, few of the DQ checks are Duplicate check Null value check Metadata check Pattern check Heterogeneous data sources Static testing not performed prior to test execution Schema validations not done Sampling strategy is used causing incomplete coverage of testing Exhaustive testing not done due to lack of automation QA Challenges Huge volume of information coming in DWH How much history to store in data warehouse, storage infrastructure vs. cost and analytical requirements Consistency of data to ensure data correctness between reporting, ad hoc query and analytics Defects caught very later in the life cycle during the review of extracts and reports No performance testing done for ad hoc reports & queries E2E data reconciliation is not done from reports to source data Lack of Skilled resources, Lack of DWH competency enablement framework, Lack of dedicated DWH Research track, Lack of differentiators and accelerators 10

11 How clients are dealing with DWH Testing Challenges Market Perspective Based on market data we see that clients who don t have a TCOE working towards setting up a TCoE By implementing TCOE, huge cost savings and quality improvement are achieved by many of Infosys clients and they have been able to compress testing timelines as well DWT COE Cost effective solution Increased focus on reuse Improved data quality and availability of systems Improved time to market to meet stringent timelines requirements Effective DW&BI enables better management decisions and reduces risks Provide strategic direction for the organization in terms of tools, licensing, processes and technology 11

12 Characteristics of a Data Warehouse TCOE.Contd. Better Quality Through Data Test Strategies ( Exhaustive, Aggregate, Sampling, Risk Based Testing etc.) Building Data Quality as the practice (Metadata, Pattern, Statistical, relationship, Business Rules Analysis etc. early in lifecycle) End to End Coverage of the DW Lifecycle ( Defined DWH life cycle to ensure complete coverage in terms of functional and non-functional requirements, also end-to-end data reconciliation) Efficiency Through automated Data Testing ( ETL Validation, Data Quality Analysis, Performance Testing can be automated using in-house /market Tools) Metrics Driven QA framework (Data Quality, Data Load, Response time etc.) Centralized repository for any DWH Testing related artifacts (process documents, templates, checklists, questionnaires etc.) Knowledge and Best Practices sharing across data testing projects (lessons learnt, defect repository, inhouse tools created etc.) Keeping up with continuously evolving DW technology (benchmarking with industry standards of data testing in terms of tools, preparedness to adopt new technology, trends etc.) Centralization and better utilization of ETL/BI/DWT tools (using the strategic tools across organization will help in saving license costs, improved utilization and training requirements DWH Testing career with defined growth path ( this will motivate people to learn and grow as career path is defined) DWH Test Academy to Skill/Reskill people, perform assessment, improve technical capability (DWH Testing skill plan for beginners, intermediate and expert level, planned technical assessment to ensure improvement of skill level) Better deployment and utilization of resources (centralized control of DWH testers to be deployed in projects based on project skill set requirements) 12

13 Characteristics of a Data Warehouse TCOE Better Quality Through Data Test Strategies Building Data Quality as the practice People DWH Testing career with defined growth path DWH Test Academy to Skill/Re-skill people, perform assessment, improve technical capability Better deployment and utilization of resources End to End Coverage of the DW Lifecycle Metrics driven QA framework DWH TCOE Centralized repository for any DWH Testing related artifacts Knowledge and Best Practices sharing across data testing projects Process Tools/Tec hnology Efficiency Through automated Data Testing Centralization and better utilization of ETL/BI/DWT tools Evaluate technology trends and identify new tools for adoption, keeping up with continuously evolving DW technology 13

14 People Competency Framework and Roles in DWH Testing Level 4 People Level 1 DWH Concepts, SQL Query writing Excel macros Data validations Basic query tools and reporting Level 2 ETL testing Test Data Management Test Strategy Defect Analysis ETL&BI Tools Automation Level 3 End to End Solution usage- Estimation, Planning, Data modeling, ETL, Data validation, Reporting, Technology Trends, Appliance testing Consulting DWT, Appliance testing, Big Data, Mobile BI, DW testing on cloud, Analytics testing Continuous improvement of individual technical competency Clarity on the roles and career path ahead Awareness of what trainings to attend, what certifications to attend, thought leadership 14

15 Processes and Best Practices for DWH Testing Process Test Automation Templates and Checklists Risk Repository Defect Repository BVA Repository Competency Development Thought Leadership Test automation tools QuerySurge, Informatica Data Validation etc. Excel based tools(macros) which can automate test steps like: test case creation, query creation, data comparison etc. Reusable templates for test planning, test strategy, status reporting etc. Reusable checklists for test plan review, pre-execution checks, execution checks etc. Ready to use DWT risk repository portal, this is invaluable for test risk planning. This can be created based on our experience and can be referred to ensure all critical scenarios are covered in test planning and scripting Business Value articulation case studies repository which can be used to implement best practices across similar projects DWT specific training program for different competency levels- basic, intermediate and advance DWT tools specific training program to create tools SME Research initiatives and repository of DWT publications to keep updated on latest trends in DWH 15

16 DWH Test Metrics Process Category Direct Metrics Derived Metrics Uniqueness # of duplicate records # of duplicate records/total number of records Correctness & Consistency Completeness # of records with pattern mismatch # of fields with inconsistent data occurrence # of records with null values in not nullable fields # of records with blank values in non blank fields # of records with pattern mismatch/total number of records # of records with null values in not nullable fields/total number of records # of records with blank values/total number of records Timeliness Delay in receiving data or feed files (hours/days) # of days delay in receiving data/ Test execution duration Phase Containment Data Load Schema Validation Performance # of data quality defects caught in each phase of project # of records loaded in target # of records rejected # of valid rejects # Total number of records in source # of entities missing from defined schema # of entities mismatching from defined schema # of data type mismatches for the fields Report response time Time taken to complete End to End data load # of data quality defects caught in one phase of project/#total data quality defects caught in project # of records loaded in target/(total number of records in source- # of valid rejects) Schema validation means comparing the defined/documented database schema with the actual DB schema, PK/FK constraints also checked here % adherence can be calculated if SLAs are defined for report response and E2E data load time 16

17 Top 3 Technology Trend in DWH/ BI Tools/Technolog y Big Data drives Tomorrow s BI Enables huge storage of datapetabytes Advantage of storing and analyzing unstructured data from social networks, public domain Helps in understand and predict customer behavior can be used for cross selling of products, customer loyalty management, real time fraud detection, compliance check etc. All top BI vendors are offering big data capabilities Information on the Move Moving from wired world to wireless world with an advantage of smartphones/tablets Technological advancement created the need for having information available on the go for faster decision making, better customer service, efficiency in business processes and improved employee productivity Most of the top banks have there banking apps available on mobile All top BI vendors are offering mobile BI capability Elastic DWH in the Cloud Lower cost in Pay per use model, over provisioning leading to high costs can be avoided Expertise of building and maintaining DWH is no longer needed within the organization itself An elastic data warehousing system in the cloud would automatically increase or decrease the number of nodes used, allowing one to save money 17

18 DWT Assessment and Transformation Roadmap 1 Establish a DWH Testing Center of Excellence 2 Enhance and standardize the current DWH testing process framework for E2E Test Life Cycle by following a standard lifecycle approach 3 Implement key DWH test metrics Process Evaluation 4 Identify strategic test tools and integrate current tools to enable end to end automation. Standardize the use of automation frameworks across projects 5 Leverage TDM function for better quality and timely provision of test data Identification of transformation initiatives based on QA Assessment recommendations Categorization of initiatives into short/medium/long term milestones Develop the plan for deployment of each initiative 6 Centralized knowledge repository of any DWH project templates, checklists, test artifacts, lessons learnt, trackers, questionnaires, training material etc. 7 Centralized training academy for skill/re-skill of DWH resources, technical assessment 8 Preparedness of DWH QA organization for adaption of new capabilities/services 18

19 Benefits of establishing a Data Warehouse TCOE Improved Control on Projects Cost Saving Adapting to Latest Market Trends Process adherence & Improvement Competency Enablement Knowledge Sharing Better Resource utilization Improved system Availability Faster Time to Market 19

20 Expected ROI of DWT COE - Key Dimensions Key Dimensions Elements Metrics to track for success Typical Improvement People Improved resource utilization Resource utilization % 10 15% Reduced resource on-boarding time Time taken to on-board resource from request to 15 30% deployment Improved Competency level Technical Assessment Results- # of people moved from lower levels to higher Helps in better project execution levels Process Following Standardized DWT processes Process Compliance Index Cost of quality 5% - 10 % Re-use of test strategy, templates, best Testing Cycle Time practices, queries etc. % of reuse 8% - 10% Tools/Technolo gy Predictive profiling of defects and proactive strategies Early Validations to catch defect early in life cycle Automation of test process and execution Internal Test Infrastructure/ tool Consolidation/virtualization Defect removal effectiveness Defect Slippage 5% -10 % Defect Containment metrics 10% -20% % Reduction in Test Execution Effort Testing coverage % Reduction in license/infra cost 10% -25% 5% - 10% 20 20

21 Conclusion Data Warehouse testing is no more limited to data and report testing, it is one of the rapidly changing technology areas and organizations need to make dedicated investment to keep up with the Market trends. As per Gartner they see future of Data Warehouse as Logical Data Warehouse, real time analytics, data visualization, domain knowledge to test industry specific use cases in data warehouse it has become essential elements of data warehouse testing. The benefits of having DWT COE cannot be ignored anymore and moving to DWT COE is a path ahead for large organizations to make maximum use of the golden mine of data. 21

22 Q&A: 22