Case Studies in Action Tips for Creating a Next- Generation Data Warehouse

Transcription

1 Case Studies in Action Tips for Creating a Next- Generation Data Warehouse Sam Strum, Director of Data Services, INTTRA events.techtarget.com SearchBusinessAnalytics SUMMIT

2 What Will Be Presented Overview of INTTRA INTTRA BI and DW Landscape as of early 2013 DW and BI Principles and Roadmap How We Ended Up Lessons Learned Next Steps SearchBusinessAnalytics SUMMIT

3 INTTRA Company Profile 220,000 REGISTERED USERS 49 CARRIERS & NVOCC 1 CONNECTION TO WORLD S LARGEST SHIPPING NETWORK 130 COUNTRIES 530,000 CONTAINERS PER WEEK 1.65m SHIPPING CONNECTIONS 109 ALLIANCE PARTNERS SearchBusinessAnalytics SUMMIT

4 INTTRA Value Proposition

5 INTTRA Information Content Types Find a Voyage: Search over 12.5M schedules from 30 of the world s most desired carriers through 1 connection. Book Your Cargo: Book directly from your search results - no rekeying involved! We transfer your selected schedule directly into your booking or if you know your vessel voyage, you can enter booking information online. Submit Shipping Instructions: Shipping instructions are automatically filled when you enter your booking details. Our flexible design enables you to start the shipment process at either booking or shipping instructions, and still automates the rest of the shipment process. Proof & Print Bill of Lading: Get faster delivery of Bill of Ladings (B/L) and minimize container shipment delays. Your B/L is always available whenever and where-ever you need for fast and easy access. Eliminate manual rekeying through the entire process and increase B/Ls quality. Track Shipment: If you enjoy logging into multiple systems to check the status of a shipment for your customer, you won t like INTTRA. We give you a single way to look up all your container status events. Manage Invoice: Now that you ve got a handle on reducing your own paper, you can set your sights on eliminating other people s paper too. Receiving invoices electronically saves time, money, and eliminates errors that can be introduced with manual entry.

6 Now that you know so much can you think of Analytical Use cases applicable to INTTRA? 1.? 2.? 3.? 4.?

7 INTTRA Business Intelligence & Data Management History Historically a transaction-focused organization - Primary revenue driven from transaction/container volume - An E-Commerce company Market share penetration has revealed an increasing opportunity to exploit the value of the data captured in those transactions - Cross Sell, Upsell, Targeted Marketing - Carrier Ranking, Growth Opportunity for Carriers Market growth has presented new information stakeholders such as the Financial Community and News Media Prior false starts in establishing a robust BI / Data Warehousing Program Key question entering Can we launch a successful Data Warehouse Program to enable analytics to meet these new needs?

8 Going into 2013, was INTTRA Organizationally Ready? Yes, but A clear and well communicated corporate strategy existed Support for BI/DW at all management levels Business and IT well aligned Forward momentum in creating a robust technical infrastructure including a somewhat under-used Data Warehouse appliance Many Areas of Opportunities The information demands were not supported by a data structure optimized for online transaction processing Improvements in data quality and validation processes Adding skills and skilled associates to the technical team Reducing spreadmarts and embracing BI Formulating and adopting governance and standardization

9 Deeper Dive to INTTRA s Early 2013 State Oracle DW with some Data Marts No BI Tool One external Product attempt Greenplum Appliance purchased Limited use Major performance problems Data silos Application silos Spreadmart proliferation Some nice Visuals Excel-based Created with the Panopticon visualization tool Web (Java) based Failed due to inability to justify numbers Difficult to maintain Used sporadically based on when Oracle performance issues encountered Purchased with a hardware first approach Copy of existing mart

10 Data Warehousing / Analytics Principles Develop a skilled, business-savvy and motivated BI/DW Team Implement a reusable and resilient underlying Integration Framework Create an integrated and comprehensive Information Repository leveraging the existing Greenplum Appliance Devise Revenue Generating, External Products for our customers that they could not live without Instantiate data ordered by business Prioritized Subject Areas Contribute Internal Business Value via robust analytic dashboards Adopt an Organizationally Appropriate BI Tool establishing well governed processes

11 2013 DW/Analytics Roadmap Release by Subject Area Formulate and specify internal and external analytic use cases Create and communicate BI development, promotion, support and adoption processes Allow time for the robust framework to be developed, reviewed and deployed with first Subject Area Based on the selected BI tool, determine need, approach and technique for reporting schemas / data mart layers Craft a BI organization to execute and grow Load to a Unified Data Layer (UDL) via a Staging Layer Perform a BI Tool evaluation then purchase, deploy and adopt the winner

12 Where Are We Today? Data Warehouse DW Technology Established 5 Subject Areas in Production Key Volume Metrics Mostly Oracle based source systems Oracle Data Integrator/Change Data Capture/flat files Greenplum load utilities Greenplum functions Booking Shipping instructions Bills of Lading Track and trace status events Invoicing 160 Tables 420 Million track and trace records (400 days) Robust Framework Unified Data 105 Million bookings (3 years) Reusable change data capture process Includes audit and monitoring Changes captured hourly Processing up to 1.2 million status events a day From 2 booking systems From 3 shipping instruction systems

13 Where Are We Today? Business Intelligence Tableau BI Tool selected, purchased, deployed and trained on Less heavy (including cost) Reasons for selection Direct and Extract Licensing Visual Intelligence Ease of Use Enterprise Capabilities (with some annoyances) 80 Server Users 10 Desktop Developers Defined Iterative Development and Promotion Process Met challenge of working around semantic layer shortcomings Multi source access including spreadsheet and can blend 4 Tier Support: First Line Departmental Enterprise Vendor 3 Tier Development: Some Self Serve Departmental Enterprise

14 Analytic Successes So Far PERFORMANCE DASHBOARD Carrier. Customer and Sales Rep Perspective Key Metrics Container Counts User Locations Network Pairs Per Product, Region, etc. Link to Sales Force Account page TnT DATA QUALITY SCORECARD Measures Completeness, Accuracy and Timeliness of Status Events Now know who has the best quality in Status Event delivery Key Learning - Certain Carriers never sent any Status Event types (e.g. Vessel Departure) External Potentially Revenue generating TRANSPORTATION MODE SHEET Facilitates effort to increase Door to Door shipments Precisely match to Shipments Includes Transshipments and Inland legs BOOKING VALIDATION REPORT Compared to existing DM/Spreadsheets Saved countless hours of SQL writing Holistic Found about 10 new DW defects Found problems in current reporting BI Leadership SUMMIT

15 Dashboard Sample Company A Company B Company C Company D Company E Company F Company G Company H Company I Company J Company K Company L Company M Company N

16 BI / Data Services Organization BI and Data Services are part of the Platform Services Department Was mostly one man with assistance from: - Project management - DW vendor partner Now expanding - Back-end / Front-end - Transform the Business / Run the Business - Internal / External Focus - Looking for Tableau expertise Administration Architecture reporting & semantic layer Developers - Data Warehouse Architect - Data Modeler

17 Lessons Learned Select the Right BI Tool Do you really need a heavy tool? Do you really need a large monolithic semantic layer? Do you need to get up and running quickly? Do you want self service and ad hoc investigative analysis from multiple perspectives? Can the tool be used externally without huge cost or complexity? Embrace, don t dissuade, end user report development Extracted, self-contained analytics: take advantage of it

18 Additional Lessons Learned Change will happen, be flexible you don t know everything up front Perform robust (even if time consuming) data modeling up front including data quality analysis Constantly test and validate and use real use cases for validating Document - it will help create end user guides for Analytics Think adoption and training from day one

19 What s Next Data Warehouse (Backend) - Additional subject areas - Enhancement backlog - Operational support processes Analytics (Frontend) - Increase Internal Reporting - Embark on External Reporting Monitoring, Evangelizing, Marketing Organizational optimization

20 Thank You! Questions? Follow ups? Reach me at Also contact and connect with me on LinkedIn at: