Evolution of the Data Warehouse at the Russian Railways. Igor Movchikov, Head of R&D department

Transcription

1 Evolution of the Data Warehouse at the Russian Railways Igor Movchikov, Head of R&D department

2 Overview The DW of the Russian Railways was originally presented at SEUGI 18 This presentation enlightens the following topics: What changes have occurred in the DW during the last year What interesting solutions and methods have been developed in this project

3 What is Russian Railways? employees (core staff only) Russian Railways consist of 17 Regional Roads and operate in 11 time zones 17 regional computer centres plus the Main Computer Centre enterprises Ministry of Russian Railways makes up to 65% of freight transportation and 70% of passenger transportation of the country

4 Why is a Data Warehouse? A lot of data sources covered different areas: cargo transportation railroad cars location etc. located in different places: legacy systems ADABAS DB2 others

5 Why is a Data Warehouse? Time - changing dimensions: stations names organisational structure etc. Impossibility to explore data from different data sources in one place from a historical view Need for analytical information

6 What does the DW give us?

7 Architecture of the DW Mainframe S/390 SUN-server NT-server Data Sources Detailed and aggregated data Data Marts Legacy Transportation Dictionary tables ADABAS DW Charges Customers Operational data queries Detailed data queries Aggregated Data queries Client computers

8 Subject Cargo transportation The first subject of the DW Contains information on freight traffic (weight, distance, cost etc.) by cargo type, stations, countries, customers and so on, down to a concrete transportation Since the launch of the system the subject has been constantly growing according to client s demands.

9 Subject Cargo transportation Detailed data: since 1996 more than 200 Gb Aggregated data (including MDDBs): more than 50 categories + 6 analysis vars about 100 Gb Daily loading: about thousand rows of detailed data entire process takes minutes

10 Forecast of cargo transportation Is necessary to increase the quality of clients servicing Predicts transportation weight and cost by cargo type, arrival and destination

11 Forecast of cargo transportation Time series is divided into several parts: trend seasonal component irregular component A statistical model to predict a trend was developed using SAS/ETS and SAS/STAT

12 Forecast of cargo transportation Trend Seasonal component Forecast

13 Forecast of cargo transportation Further development: Provide analysts with an ability to interactively create statistical models, modify prediction interval and so on Use sophisticated statistical analysis to improve quality of the model Add external economic indexes (currency exchange rate and other)

14 New subject Railroad cars Contains information on locations of railroad cars and operations with them (loading, unloading, repairing and so on) Is highly integrated with the subject Cargo transportation

15 New subject Railroad cars Necessary data in an operational database (ADABAS) is spread across several tables Extraction and transformation process is very complicated, it looks like this:

16 New subject Railroad cars Performance attributes: As a whole about 1.6 million railroad cars Daily feeding: thousand rows of detailed data 70 Mbytes of detailed data entire process takes hours More than 80 categories, 3 analysis variables

17 New subject Railroad cars Several analytical tasks are being developed on the base of this subject: optimisation of cars movements to reduce transportation expenses and delivery time optimisation of empty cars usage to reduce empty cars transportation co-ordinated cargo delivery for large-scale clients

18 Solutions and methods System of development organisation: Development environment Applications Data + WA Metadata Exploitation Assembling tool DW Loader tool Loading Distribution Data + Metadata Auto-update tool... Clients DW Loader tool Operator

19 Solutions and methods Assembling tool - helps a developer in preparing a distribution package of a client application Auto-update tool - automatically downloads a new version of a client application and installs it on a user s computer

20 Solutions and methods DW Loader key component of ETL process: SAS/WA add-ins, developed in SAS Russia Enables to load the whole DW or one of its part Exports WA metadata into a production environment Controls execution of each step Monitors the results of the execution Can be utilised by unqualified users like DW operators

21 Solutions and methods Centralized journal of user s queries some modifications have been done to a standard HOLAP journal Different reports based on the journal monitoring and controlling of users activity analysis of high frequency queries for their optimisation

22 Further development Addition of new subjects, first of all financial information Development of analytical applications according to users demands. Adaptation of client applications to the WEB-technologies

23 Conclusion The DW has been constantly growing up because it is exactly what users need It is the SAS system that enables developers to rapidly solve new tasks Consultants of SAS Russia have been giving high-quality technical support and consulting for the whole duration of the project

24