ARCHIVING - PURGING IN DHL Krishna Sabbineni & Anand Vasudevan Xchanging/DHL March, 2014
AGENDA DHL Background & OTM Usage Business Case & Challenges Options Explored, Timeline & Scope Archiving Solution Design Implementation & Design Components Web Solution Screens Migration Strategy to new version Post Go-Lives Issues & Challenges Q&A Session 2 Archiving @ DHL An insight
WHERE WE USE OTM WITHIN DPDHL? Archiving implemented for this OTM setup in DGF 3 Archiving @ DHL An insight
BUSINESS CASE! Requirement to archive data after six months.! Ability to do a track and trace on purged data.! Ability to run reports on archived data.! Minimal impact to the Production environment! Solution to be fast & highly reliable 4 Archiving @ DHL An insight
OTM @ DGF Xchanging offers the below OTM servcies:- Key services are:! Order Management & Visibility! Optimized Transports! Improved Performance! Traceability & Alert Management OTM Implementation Custom Development 24 x 7 Support and Maintenance Version Upgrade! Individual care! X-Dock and Merge In Transit! Data Archiving and Reporting SCI (OTM) tool is the key application to support the service delivery towards the customers. SCI needs to be fully embedded into the current business and to provide efficient services to the customer. 5 Archiving @ DHL An insight
DHL DATA VOLUMES! Average Inbound EDI Messages per day (approx.) Shipment Updates 3,400 Inbound Orders 10,000 Transportation Requests 3,000 Inbound EDI Transactions - 80,000 Outbound EDI Transactions 4,200 6 Archiving @ DHL An insight
CHALLENGES! Single database for multiple customers hence Rate of Data accumulation is high! Huge amount of data gets generated which needs periodic purging! Sensitive Production environment Any performance degradation causes Business Impact! Compatible with later OTM versions! OTM Migration planned but to which version?! Maintain Data Integrity! Provide a web interface that is close to the existing SCI branding 7 Archiving @ DHL An insight
PRESSURE POINTS " Existing customers demand archiving and reporting and even linked it to contractual obligations " Customers and users request a high performing application e.g Cust A requests a response time within 4 seconds " Future strategic business development clearly demands for the Service " Flexible and customisable reporting capability for dedicated customers! Need for one Archiving environment " Current workarounds to keep data longer than 6 months are very complex to manage " Performance issues due to heavy workload on the operational database " Bad data quality caused by inconsistent purging " No structured and synchronous information exchange/replication 8 Archiving @ DHL An insight
OPTIONS EXPLORED! Traditional Approach: Using Standard Oracle functionality to create a Real-time Operational Schema Recycling primary keys Performance issues Migration Front-end will be an OTM one but needed a leaner front-end Archive solution should work seamlessly during version migration where migrations are done in batches 9 Archiving @ DHL An insight
ETL TOOLS CONSIDERED GENIO/GOLDEN GATE & CUSTOM BUILD SCRIPTS Key Factors which influenced the selection of GG! Robustness of the solution! Complexity of Set-Up/Maintenance! Prior experience of some of the tools! Compatibility/Support with OTM! Availability of Resources 10 Archiving @ DHL An insight
TIMELINE & SCOPE Initial talks and presentations conducted in May 2011 BRS and first cost proposals: Signed off: 30 July 2011 Proof of Concept Signed off: 28 August 2011 Final Solution Completed (5.5) June 2012 SCI Version Migration First customer on 6.3 June 2013 Archive Migration Initiated August 2013 Archive Migration Completed January 2014 SCI Version Migration Completed February 2014! Data Archiving Solution! Web Front End to view data 11 Archiving @ DHL An insight
SCI BUSINESS OBJECTS FOR ARCHIVING! Order Base! Order Releases! Order Movements! Shipments! Invoices! SKU 12 Archiving @ DHL An insight
SOLUTION DIAGRAM - OVERVIEW 13 Archiving @ DHL An insight
DESIGN LOGICAL COMPONENTS! Replicated Online Database ROD! Non-OTM Front End! Golden Gate - allows operating, managing and controlling data processing on each server! Golden Gate Director - a stand alone application orchestrating all Golden Gate components spread across source, target and any other intermediate servers. It is a single point to Administer and Monitor SCI Archive data processing 14 Archiving @ DHL An insight
ARCHIVING TOOL - GOLDEN GATE! Provide close to real time data replication for ROD database.! Does not cause performance impact on OTM production database. Golden Gate uses transaction logs for data extraction from the source database, so does not need to run any queries on production data.! Perform online data transformation before loading into Archive databases. Transformation can be performed on target database server 15 Archiving @ DHL An insight
REPLICATE OPTIONS Top Option Chosen due to the size of database 16 Archiving @ DHL An insight
IMPLEMENTATION DATA SIZE! Daily Redo Data size 15-30 GB/day! TMS Database size = 320 GB (approx.)! Initial Data Load and REPLICAT Option chosen! Golden Gate Director Basic Monitoring View process information and statistics, including data latency between source and target databases View GoldenGate process log Configure e-mail notification about alerts 17 Archiving @ DHL An insight
DESIGN DATA FLOW DIAGRAM 18 Archiving @ DHL An insight
WEB SOLUTION SCREENS LOGIN PAGE 19 Archiving @ DHL An insight
WEB SOLUTION SEARCH CRITERIA! Search window popup Search window popup comes when clicked against any field! Search criteria help Begins With : Searches for records with field text that appears at the beginning of the text string. Ends With : Searches for records with field text that appears at the end of the text string. Is Null : Searches for records with field text that is blank. Not Null : Searches for records with field text that not blank. One Of : Finds a series of specific values. Not One Of : Eliminates a series of specific values from the search. Contains : Searches for records with field text that appears anywhere within a text string. Same As : Searches for records with field text that is an exact match. Not Same As : Eliminates field text to be considered in the search. 20 Archiving @ DHL An insight
WEB SOLUTION- SHIPMENT SEARCH! Search Shipment Enter any search criteria for Shipment and click Enter Multiple Tabs (Shipment, Order Release, Status, Advanced) on which the search criteria can be applied 21 Archiving @ DHL An insight
WEB SOLUTION SHIPMENT VIEW! View Click on the hyperlink of a any particular Shipment in the Result screen which need to be viewed in detail 22 Archiving @ DHL An insight
MIGRATION FROM 5.5 TO 6.3! Parallel Project for TMS Migration 5.5 - > 6.3 with a finish line of 6-8 months! Not all customers use Archive solution! Current TMS stores data for 6 months data! Customers connect to ROD for data older than 6 months 23 Archiving @ DHL An insight
POST GO-LIVE SUPPORT! Periodic monitoring of the application by one of the monitoring technique provided by the Golden Gate application tool! Random checks to ensure that replication is being done correctly! Cross-checking selected data for key transaction tables Orders, Order Release or Shipment tables! Counting the number of row of random or key tables 24 Archiving @ DHL An insight
ISSUES AFTER GO-LIVE / CHALLENGES! Old Shipment updates being received OTM creates a new record which means a duplicate entry in Archive database! Foreign Key Constraints Locations or Contacts! Missing Archived Logs in case of any break in Archiving. The rate of Archive ReDo logs creation is quite high and the restriction of disk space means backing up to tape drives for later retrieval is complex! Experienced Golden Gate Resources 25 Archiving @ DHL An insight
NEXT STEPS! Include all OTM modules - all set of tables! Include the SCI VIS Solution database! Include the BI reports to be run from BI Publisher 26 Archiving @ DHL An insight
XCHANGING OVERVIEW Our Core Principles What We Do Xchanging provides business processing, technology and procurement services internationally for customer across multiple industries. What We Are Xchanging brings innovation, thought leadership and passion to its customers business so as to enhance performance and value. Our Values are embedded into everything we do. What We Want To Be Xchanging wants to be regarded as the best provider in its chosen markets by delivering services that are recognised for outstanding quality, reliability and innovation. Our Global Presence * 8,000+ employees Customers in 48 countries Industry Sectors that we Serve Servicing our customers 27 * Upcoming site Aerospace & Defence Automotive Banks & Financial Energy & Utilities Services Food & Beverage Healthcare Insurance Real Estate Retail Logistics Manufacturing Telecommunications FMCG Public sector Transportation, Travel & Leisure 2013 Xchanging plc
TECHNOLOGY SERVICE OFFERINGS Support & Infrastructure services L1/L2/L3 support, Backlog reduction, IV & V Application Performance Management, Customizations, Upgrade, migration, security, patch & Incident Mgt Business Consulting / Strategy Product strategy, Consulting, Business requirements, Applications rationalization, Technology evaluation, System Architecture, Product Roadmap, budgetary assessment, Proof of Concept, Business validation Application Implementation Implementation of enterprise application software such as Oracle Apps, OTM, SAP Workflow / database management systems End to end Industry specific suites ERP, BPM, EAI and b2bi software Tools IP Processes Methodology Development to Production Product Requirements Analysis & Definition Product Architectural Design, Detailed Design, Product Quality Management, Test Strategy, Test Automation, System & User testing, Release Planning, Product deployment, Acceptance Genius IRIS TESA XMM ELGAR Brokasure CRONUS NetCommex Application Development Requirements Analysis & Definition High Level Design (HLD), Low Level Design (LLD) Build and Unit Testing Test Strategy, Test Automation, System testing, User Acceptance testing & Support Deployment, Go Live 28 2013 Xchanging plc
QUESTIONS 29 PRESENTATION INFORMATION IN FOOTER
THANK YOU