A Work Method to build Data Warehouse Applications Niklas Karlsson, Niklas Data Europe BV Introduction The purpose with this paper is to discuss a method that has been used extensively at Niklas Data for a couple of years aiming to build applications based on SAS. The method has been slightly modified during the last year to adopt it to the new major trend, Data Warehouse applications. A Data Warehouse is a major investment in a company's future due to its focus on new business opportunities as opposed to controlling business retrospectively. Our experience tells us that is not possible to purchase a software that is the Data Warehouse package, even though a number of software vendors are giving that impression. Your business is unique and it demands unique applications to support your current business strategies and to support the creation of new business opportunities in the future. NikiasData Niklas Data is an independent SAS consulting company owned by the founders and management. Since the beginning in 1987, our vision is to be the premier provider of solutions based on SAS. In 1994 we were the first consulting company in Scandinavia to be certified as a SAS Quality Partner. Currently, we have offices in Sweden and Norway and the Netherlands. Our head office is situated in Netherlands and during the next five years we have plans to open offices in Denmark, Finland, Belgium and Germany. We are unique in many aspects but the main one is that we only work with SAS. Niklas Data employs more than 50 consultants specialised in building SAS applications for organisations and corporations. Our plans are to grow to more than 100 consultants during the next year, due to the large demand of expertise in the areas we specialise in. Some examples of solutions we have helped our clients with are Data Warehouse, Executive Information Systems, Quality Statistics applications, Analysis applications, Forecasting applications and numerous different statistical applications. Why a Data Warehouse? During the last years a quite a few companies have run into severe problems because of management lacking the information needed to make the right decisions. The problem is usually not that1he inforination is missing, rather than the information is not available to management in a format that they can draw the proper conclusions from. At Niklas Data, we use the Data Warehouse concept in this case as a tool for Risk Management. Because management is missing the data, they are loosing new business opportunities. Most businesses today are spending a considerable amount of money controlling their business based on legislative demands through the financial systems and inventory systems. A recent 219
survey in USA showed that a companies spend one percent on acquiring business, 9 percent to do business and 90 percent to keep score of the business. With a Data Warehouse we open up ways for management to spend more time on getting business and lesser time to keep score of the business. If the benefits with Data Warehouse are so evident, Why have we not built one before? The business community has tried for many years but we have always had problems with the data, because all data is usually on clerical functional level suited for the old production systems. The problem is that we are lacking corporate level data, that meets the demand of top management. Usually, we also lack the historical data because the clerical level of production system do not need any historical data older than say 30 to 90 days. In order to be able to make the proper decisions, business management needs a lot of historical data, as much as they can get! During the last years we have experienced a trend in moving from functionally oriented organisations, supported by transaction systems, with one system per function in the company, to process oriented organisations, supported by transaction systems, with totally integrated solutions for all functions in the company. Many companies are buying these new huge process oriented transaction systems and getting rid of their old functionally oriented transaction systems, to a great cost. A Data Warehouse is an alternative to investing in new totally integrated software packages or to rebuild everything you have for all your business. A Data Warehouse can be seen as a tool for steering your company by processes instead of by function, without huge investments in completely new systems. Since the Data Warehouse concept is being used in several contexts, in order to avoid any confusion we will present Niklas Data's definition of a Data Warehouse. Niklas Data' s definition of Data Warehouse Most of US have heard Bill Inmon's definition of a Data Warehouse, thafit should be subject-oriented, non-volatile, time variant and integrated. At Niklas Data, we have a more pragmatic picture of the Data Warehouse. We see the Data Warehouse as the tool needed by business management to do their job. The data in the Data Warehouse must be adjusted to the needs of management, one of those needs is that the data should be possible to be viewed from the process perspective of from the function perspective. An example of this perspective can be management's need to see everything about a satisfying a customer, regardless which products or services this customer buys. That perspective creates a need to fetch data from several different production systems, that one and each support a single function in the organisation. Systems that need to be accessed can be order systems, perhaps from different product areas, production systems, warehousing and finally financial systems. After completing this cumbersome data fetching activities there is a need to define customer, in order to not to compare apples with oranges.,fpllowing this there is a need to transform the data, clean the data, summarise the data and maybe some additional operation before he has an answer for his probably quite easy ques.tion. Of course, today most companies are not able to perform all of these tasks and if they aie it takes a considerable effort, both time and resource consuming, one of the main reasons for developing a Data Warehouse. Us from the IT-department help management fetch the data, organise it and store 220
it. Whenthis is done it should be quite easy for management to do their job asking queries against the database. Needless to say, a Data Warehouse solution cannot be developed without extensive participation from management and end users in the initial phase of the project. Corporate Data Warehouse is the term we use when referring to all parts of system support for management. It includes routines for accessing production data, fetch data, transform, clean the data, and load it down to the Data Warehouse. When the information is in the Data Warehouse on a detail level we create Data Marts that are subsets of the Data Warehouse. catering for one specific business process needs. The reason for creating Data Marts is mainly performance but also security. In the definition of the Corporate Data Warehouse we also include the tools that management uses for Decision Support, they can be Executive Information Systems, query tools, spreadsheets etc. Corporate Data Warehouse Production environment New opportunities Management level Clerical level SAS as the complete Data Warehouse solution Over the years we have built a number of applications based on SAS that fetch, analyse and visualise data for management. The difference between these applications and the concept of a Corporate Data Warehouse is that now we talk about providing support to the entire corporation. We do not build islands anymore. We use the same Meta Database, we use the same and defined terms for variables in the applications and so forth. One major reason for the growing success of Data Warehouse solutions is that it is now politically correct to do it because both the management and IT -department sponsors the solution. Of course, they do it from different perspectives. Management needs the data to stay ahead of their competitors and the IT -department needs it to take live up to the service levels they are working under and as a benefit the IT-department will regain the control of the DSS-applications and their data. At Niklas Data, we see SAS as the only integrated tool for both building and managing a Corporate Data Warehouses. SAS can access almost any datastructure that is available in the market. It is a fairly easy task for us to transform and clean the data based on information in the Meta Database. When the data is cleaned, SAS is a very powerful tool for loading data 221
down to the large Data Warehouse database. Once in place in the Data Warehouse we extract subsets of the data into smaller databases and during the copy we aggregate the data and derive new variables. The reason behind this is that the applications and queries used by management must be fast and effective. A second reason is security, it is a lot easier to control securities in the "small" Data Marts than in the Data Warehouse database. Corporate Data Warehouse Production environment Selection Summary Derive Transfonning Cleaning Loading SASIACCESS The Nildas Data Work Method At Niklas Data we always work in projects and always in partnership with our clients. Most projects that we are involved in are mission-critical for the client and demand support of both good project management and developers. To help the project to reach their client demands we works closely with the client and we works with routines for Change management, documentation and Quality Assurance. When we are working together with clients building Data Warehouse or other types of Decision Support applications we need to use methods of Rapid Application Development, because otherwise the system we are building will be old before it's even ready. The reason for that is that in this hard business reality out their, the companies are fighting every day for market shares, and the way they are doing that isa constant ongoing change to their business adopted to their customers needs., I., When we start a new project together with a client we always aim to be ready in 180 days. Of course it can be hard to get everything included in 180 days, but we believe it is better to get 80 percent in 180 days and then deliver a system to help management to make the right decisions, then to deliver it after 3 to 5 years with "everything" included. Someone has to "pay the bill". Usually that is the top management and they would like to see results now (they believe that 180 days is too much). After the first delivery after 180 days we usually go into an iterative phase with deliv~ries every 90 day. We use the same method but we speed up the prototype and realisation phase. 222
Every project we start is unique and demands different skills in different phases of the project. Niklas Data usually sets up a project organisation together with the client. Project organisation The project organisations at Niklas Data are always focused on the Clients objectives. We add right resources in the right time with right competence. The project manager is 100% committed to solve the Client objectives with support from Functional Analysts Technical Programmers and Change Management resources. Available to the project manager is a network expertise and support both inside Niklas Data and through our partners. Typically, there could be issues about Communication, Technology, Experts in special parts ofsas or different functional competence. We have an extensive Quality Assurance program, in which preferably people not involved in the day-to-day work of the project conduct periodic reviews. 223
Three phases of the method As previously stated, we use a method in all our projects we are committed to deliver an application to the client in 180 days or less. Our method consists of three phases. The first phase takes approximately one week in calendar time and it demands heavy' involvement from top management. The second phase is aimed at showing management, and of course the IT -department, how their business needs are solved through a SAS based applications. The third phase is the realisation phase which is more traditional systems development phase. The three phases are named: 1) Information Gathering 2) Prototyping 3) Realisation 1, 2 3 180 days The Information'Gathering phase This phase is short, very short. In this phase we have heavy management involvement, in order to clarify the company's vision and business strategies. When that is completed we select one or two main business process to focus on. For each business process its key performance objectives are identified. During this phase we work hard to try to identify company's "million dollar question". We seek to find out the most important decision support problem, and focus our initial efforts on solving that. The reason is that we quickly want a success story to secure sponsor commitments from both management and IT-department. This phase thoroughly documented in an Information Gathering report. The report along with a suggestion on which process we should focus on in the following phase. We also deliver a cost estimate for the next phase and for the rest of the project. We try to give a cost benefit analysis of the entire project. The Prototype phase, \ Based upon the Information Gathering report we start the Prototype phase by defining criteria's for the prototype. Time is crucial in this phase. We usually have two to three weeks of calenqar time to finalise and deliver a prototype, on which they base a decision about the next phase. 224 i j
, ', It is in this phase that we start to look at the technical architecture for the application. We let our most experienced consultants build the prototype in close co-operation with the client. It is very important that the sponsor, whose commitment we secured in the first phase, is working close together with the development team and that we keep in focus our ambition to solve "the million dollar question". During this phase the project manager finalises, together with the senior consultants, the project plan for the rest of the project. An important task in this phase is to limit the scope of the project. Since we are committed to deliver the application in 180 days it is important that the sponsor, the IT-department, and the project members can reach a consensus about a "good enough" application.. This phase ends with the delivery of the prototype and the finalised project plan. The Realisation phase The realisation phase resembles most traditional IT-projects. We are still working close together with the client, even if we now start to work more traditionally with systems design, data modelling, program design, programming, and documentation. When the system development is ready, we go into an extensive system and acceptance test period to secure that the application is functioning the way it has been specified. We are working with the quality of not just the application, we also work hard to secure that the data for the Data Warehouse is transformed, cleaned and adjusted for Decision Support. During the pilot operations we let the knowledge workers at the client who are the primary users of the application work with the application. The purpose is to identify any severe performance problems. Another purpose is to identify any typical questions. for this user group. I typical, repeated questions are identified, we add a new table to the Data Warehouse, pre-summarized and with exactly the columns the user would like to see. This is the first time we deliberately start to denormalize the database and the reason is primarily performance. At this stage we increase our change management efforts. We develop a communication plan, a roll-out plan and draft the curriculum plan. The "marketing" effort of a solution of this magnitude should never be under estimated. Our experience is that one of biggest risks with this type of application is that the users are not being given the proper amount of training in the proper time. Another task is to start up an help-desk function to help the customer with answers on questions that arise when they use the application. The help-desk can also help the users with complex queries, and if the query is coming back time after time they can add it to query-toolbox that is available within the application. Next we move the application to production. When the application is in production we start to work intensively with training and roll-out of the system. 225
Iterative process When we have delivered the application, aiming at solving the "million dollar question" in 180 days, the work is not completely finished. We now start up small projects with deliveries every 90 day period. The reason why we have shortened the time frame from 180 days to 90 days is that the first 180 days included a lot of work defining the Corporate Data Model, Technical Infrastructure, definition of the Meta Database and a lo~of other tasks e.g.. design standard, programming standard, documentation standard, which is done once and for all: 180 days 3 \ 2 (\ 1 \ \ 90 days 1 2, 3 J, / This 90 days time frame projects are usually done in sequence, but if there is a need to speed up the process to build a complete corporate data warehouse we could run them in parallel. The major obstacle to running them in parallel is a potential staffing problem, since we never compromise with having the right skills. For a major corporation this will be an ongoing effort of running small 90 days projects because a Data Warehouse will never finalised completely. The business will constantly change and these changes need to be reflected in the Data Warehouse. When the first Data Warehouse projects has been successful and management comes to fully appreciate the possibilities to find new business opportunities they will demand changes in the Data Warehouse to always reflect the way of doing business. This is the reason why we always starts with a small project to solve a specific business objective and after that one is solved we take a new one. Even if we start small, we have to think big when we are designing the technical architecture and setting up the organisation 226
around the Corporate Data Warehouse. If the big picture is overlooked you are running a great risk of running into problems with your hardware and software, because it may not capable to handle the volumes of data the business demands. The issues pertaining to the organisation around the Corporate Data Warehouse, as well as around the roll-out, must not be overlooked. There is a need for a Corporate Data Manager as well as an organisation for education and support of the users. The users.. of the Data Warehouse is usually management and usually demand education when they can and they demand answers from the help-desk when they call. If you can't give them that they will soon end up not using the Data Warehouse. Conclusions SAS is the only completely integrated Tool for Data Warehousing. When developing a Data Warehouse it is essential to take a business perspective and try to come up with the million dollar question. Iterative development is the only way to secure a successful implementation. Realise that there is a corporate level of data and that is the perspective of management, who are the key users.. I hope that you enjoyed this paper. If you have any questions, please do not hesitate to contact me for further information. Niklas Karlsson Nildas Data Europe +468282424 227