Data Warehousing & BI for the Small to Midsize Business

Transcription

1 Cirista White Paper Data Warehousing & BI for the Small to Midsize Business By Joe Foley & Tim Bates April 27, 2004 Cirista

2 Cirista White Paper 2 Introduction Everyone seems to agree that a Business Intelligence and Data Warehouse (DWH) solution is important for business and even that it is required in competitive markets. So, obviously, every company has implemented a BI/DWH system, right? Well, if you have not yet jumped on this bandwagon, don t worry, most small to mid - size businesses 1 have not made that decision. If BI really is so important, how is it possible that most businesses have not added it to their information management systems? Cost! The cost of implementing a classic reasonably robust BI/DWH system can easily run into high six and even seven digits and the ongoing cost involves at least a few full time support people as well as hardware and software maintenance, operating cost, etc. Not only is a BI/DWH solution a major budget buster, but also the justification for these systems is often hard to identify and harder to prove. However, when your business has to compete in today s hyper competitive markets and against Fortune 1000 companies that have a full suite of BI systems, you are at a competitive disadvantage. To stay competitive, you need to be able to identify opportunities and problems early and adjust your business quickly. Which means, you need more analytical power than you can get from using Excel on your financials. First, Some Definitions Before we get into the solution to this issue, it is important to define terms. Many people consider BI and analytics to be two different concepts while just as many others see them as very closely related, if not identical, terms. Others see a spreadsheet and some data as the definition of BI. The Oxford English Dictionary lists three definitions of analysis that would seem to relate to business analysis. The first states that analysis is the resolution of anything complex into its simple elements. The second definition is a tabular statement, a synopsis or conspectus, of the results of the above process. In the third, analysis is defined as the tracing of things to their sources; the discovery of general principles underlying concrete phenomena. 1 Small to Mid-Size Businesses (SMB) are defined by the number of employees within the company or total revenue. For the scope of our discussion, we will use revenue as our criteria and consider this group to be those companies with revenues of 500 million dollars or less.

3 Cirista White Paper 3 Business Intelligence is the resolution of masses of complex business activity data into actionable information elements. This is often presented as a tabular statement resulting from the above process. Generally, BI is the result of tracing business results back to the detailed source data resulting from business operations. It seems that there may be some similarity between analysis and BI. There is also a great deal of confusion over the definition of BI caused by software sellers using any currently popular term to describe their products. There are reporting systems that are being sold as BI software. Data Mining is often called BI. OLAP, MOLAP, etc. are all considered BI by many people. Some people would say their data warehouse is providing all of the BI that is needed. For the purposes of this paper, we will define BI as providing a combination of data exploration and analysis to business decision makers in support of their business activity. This description does not exclude a data warehouse, data mining, reporting, etc. but it does go beyond each of these specific technical products. This definition means that you will not be implementing BI by simply building a data warehouse, installing an OLAP tool or acquiring a data-mining tool but instead looking at the whole business issue and, more importantly, the result. When a business is simple, simple reporting of facts is sufficient for BI. But, for most small to mid size businesses today, even all of the tools listed above are not sufficient for the variety of challenges that are faced daily. The key attribute of BI is that it provides the intelligence to support a business activity. This is not a technology-based attribute and consequently, there is no good or complete technology based definition of BI. The business may be anything from managing Dollar stores to sending rockets into space. The activity may be anything from strategic planning, to audit, to stocking shelves with the right merchandise. Obviously, the correct business intelligence for any situation can have a very wide range of specific technical definitions. Where is the value of BI found? Since it is so difficult to define what BI is, a better approach may be to define the return on investment that will be achieved by data exploration and analysis. Unfortunately, this is just as difficult as defining BI. One industry expert says A data warehouse offers no tangible return on investment to an organization. The value lies in the processes that the data warehouse enables. 2 If this is true, how can you define the expected ROI of such a project and how can you measure the results once the project is completed? If a person makes a better business 2 Joe Leudke, BI for the Masses, DM Review Online, March 2004

4 Cirista White Paper 4 decision, is it because of the BI project or because the intelligent person is always learning and improving their skills? If you decide to credit the BI project with helping to make a better decision, what share of the benefit is a result of the BI that had been provided and how much is due to superior individual performance? The answer is very subjective buy still should be answered as honestly as possible. If the BI system does not have enough payback, don t keep throwing money at it. One excellent way to ensure that the system has a positive return is to ensure that it has a very low up front cost. Then, even a small return will provide a positive ROI. Also, this makes it easier to kill a non-producing project. Very few people want to drop a project after they have invested hundreds of thousands of dollars. How Much BI/DWH Can You Afford? If you have determined that you need a BI/DWH solution to stay competitive in your business, one of the first questions that must be answered is What is the budget? The next question should be When is that budget needed and what is the ongoing budget requirement? The true cost of a project is the total of the implementation costs (hardware, all software, staff, training, consulting) plus the ongoing operating cost (support charges, operating staff and user costs). Gartner forecasts for 2004 an average expense rate for all IT in an organization at about 3% of revenue. Of that, certainly no more than 10% can be used for BI/DWH systems 3. For a typical $100 million business, this means no more than $300,000 per year can be allocated to BI/DWH system development, implementation, operation and maintenance. Implementation costs using a conventional approach vary widely depending on your selection of products and the scope of the project. However, it is safe to say that the barest minimum cost will be more than $100,000 and the maximum can be well into seven digits with a typical cost probably over $500 thousand. If the total cost for our typical $100 million company exceeds $300,000 then the rest of the cost must be carried over to a future period. If the new system for our $100 million company will require three FTE s to support it and use it then, with hardware and software support costs and ongoing training and other costs, they are using up everything that they can afford just to operate the system. Either they can buy the system and never use it or, they can afford to use it but can t buy it. Neither option looks very attractive. 3 We are using Joe Leudke s stated position on BI investment amount for SMB s. BI for the Masses, DM Review Online, March 2004

5 Cirista White Paper 5 A Workable Solution With all of these problems with defining what BI is and how it will help the business, it is no surprise that few small to mid sized businesses have started down the BI road. However, that does not mean that they should, or even can, ignore the BI issue. Somehow, each business must be able to compete effectively with the biggest players in their industry or they will disappear. To be honest, the smaller company will usually need to be better than the big competitor. If they are equally as good, the big guy will usually win. A good workable BI/DWH solution for the value oriented, small to mid size business, will have some common characteristics. The most important of these is that the solution must be cost effective. The implementation cost and ongoing operating cost must be within the budget range of the organization and the return on that investment must be substantial and earned quickly. Second, the front-end risk must be low. That is, the cost of entry must be low and the time to payback must be short. The small to mid size business can risk some investment to build for the future but those investments must quickly show value or be stopped before losses become too great. The third common characteristic is that the system must be able to grow with the needs of the enterprise. It would be far too costly both for implementation and operations to use new software every time a new BI need arises. Even when the software and hardware are virtually free, the operating cost is too high. The ordinary spreadsheet is a wonderful analytical tool but attempting to provide enterprise wide, coordinated BI with dozens of spreadsheets would be impossibly difficult and expensive. The workable BI solution for the small to mid size business will be implemented at a low cost, have a quick and highly valuable return to the organization, will have low operating costs and will scale to cover all enterprise BI/DWH needs. The Cirista BI/DWH Solution A BI system that can start small on a single project and grow in a coordinated manner to a full enterprise BI/DWH system would meet many of the requirements for the small to mid size business. Start costs would be low and the payback would be quick. This is the Cirista approach to enterprise BI. A comparison of the cost and payback of the Cirista approach and the conventional system implementation is seen in the chart below.

6 Cirista White Paper 6 The conventional approach requires a large front-end expense to purchase all software and hardware and to implement the full system. Then, after all of that front-end expense, the return on that investment cannot start until after the system is completely implemented. Even with a strong return, the conventional system may never show a positive ROI. The Cirista approach combines a lower overall cost of implementation with an incremental installation plan. This method defers part of the cost until a later time and brings the first ROI in sooner. The net effect of this is that there is very little time or money put at risk, regardless of the scope of the final system. If the system does not provide positive ROI early in the project then further expenses are killed before they become a serious drain on the business. On the flip side, even a small business improvement will cover the cost of implementing and operating the system and provide early and positive ROI. Clearly, the conventional approach may or may not ever provide a positive ROI. If it does, the gains will be far into the future. However, the Cirista approach provides a break-even very quickly and an ROI that is clear, easy to measure (even with the most conservative assignment of value to the BI system) and steadily growing. Even quite small companies can afford this approach since there is so little resource at risk at any point in time. Over time, the cost difference between the conventional BI system and the Cirista approach grows to be very large. This is due to the fact that the largest component of the cost of those systems is the labor cost associated with operations over time. With just a.

7 Cirista White Paper 7 DBA, an SQL programmer and a system administrator, added to the cost of the users of the system, the operating costs rise very rapidly. With Cirista, the ongoing cost will primarily be the business users. Even a very large and intricate installation would not require more than one person to manage all of the technical aspects of the system. The most import difference between a classic BI solution and Cirista s is the relative costs of the parts of the installation. Even if all of the supporting hardware and software were free, the conventional system is more expensive than the entire Cirista solution. This is due to the extensive efforts needed for implementation and the high cost of ongoing operations with conventional BI systems. Unfortunately, these ongoing costs are the ones that are most often overlooked when budgeting for a BI system. The other point that should be noted is that there is no DBMS in the Cirista solution. The Cirista products have their own data storage system that is designed specifically for analytical processes. This not only saves you money, it provides analytical capability that cannot be achieved when using any relational database to store the data. The Cirista approach can start as small as a single user system installed on a desktop computer. Total implementation cost, including hardware, software, training, staff and consulting could be less than ten thousand dollars. Ongoing costs, including most of an FTE using the system, would be less than fifty thousand per year. That is easily within the budget of all but the smallest companies. As this basic system is scaled up to meet broader enterprise requirements, the increased costs will primarily be the additional user time resulting from having more business users on the system. Fortunately, this is the portion of the cost that is most closely associated with providing a positive return to the business. The IT costs in operating the system would primarily be related to the number of unique sources of data that were used and the number of users of the system. However, the installation would have to be very large and complex before one full-time person would be needed for IT support. Return on Investment? Unfortunately, all of these plans and schemes and technical features have no value if the system does not provide the expected business value. To provide this, especially in the small to mid sized company, the system must be: 1. Easy enough to implement so that IT staff requirements are minimized (No more than 1 person.) 2. Easy enough to use so that any business person can use it quickly (No programming or cube building, ever.) 3. Powerful enough to provide the intelligence that is needed (OLAP, Data mining, and other like tools.) And, to provide ongoing value,

8 Cirista White Paper 8 4. The system must be scaleable enough to serve as the enterprise data warehouse. All of the planned and projected ROI does not put one cent in the bank. The system must deliver actionable information to the right person at the right time or the BI investment becomes just a wasted expense. The Cirista products are designed specifically for these requirements. There is no SQL or other programming language required or even provided in any part of any user interface. There are no complex design issues like dimensionality of data, normalizing or denormalizing, indexing, aggregating and so on. These are the tasks that make the typical implementation project last for months. By eliminating the need for these steps, the startup, which is all expense and no payback, is reduced with the Cirista solution to a small fraction of the expected time. To dig the intelligence out of your data you just load and go. Cirista implementation projects are measured in days not months. For optimal ease of use, the various BI user types (power user, operational people, executives, etc) each have an access tool that is designed for them. For all of these user types, access is through point-and-click, drag-and-drop interfaces. Another important aspect of ease of use is response time. To provide a business user with value, that user must be able to focus on the issues until a resolution has been reached. When analyzing a difficult issue, breaks as short as a few hours to write a new extract or build a new index, will destroy the users line of thought. A break of days or weeks will usually mean that the answer will never be found. Every question must be answered directly (if the data exists) and promptly. This will allow a businessperson to focus on business issues rather than technical issues. The Cirista products respond to every question (where the data exists) with no breaks for any purpose. As quickly as the computer can assemble the required data into a usable form, it is presented. Due to the unique storage method and the powerful analytic tools used in the Cirista products, this will usually be as fast as a user can respond to the information, generally, a second or two. Because it is possible to be analyzing the contents of many millions of records, it can take as much as a minute or more for an exceptionally difficult analysis, but there is never an extended break to write a program or fix the structure of data. - labor requests that the SMB does not have the luxury of making. The businessperson can work toward a resolution as fast as they are capable of thinking through the questions and answers. These features provide the user with an easy to use and responsive BI system. However, a good video game is also easy to use and responsive. The BI system also must have unbounded analytical power to provide the promised return on investment. The installation, training and setup are all investment. The analysis is where the return on that investment comes from. With features like auto navigation, segmentation, dynamic data filters, n-dimensional drill reports, The Cirista products provide powerful analytical tools that make business

9 Cirista White Paper 9 intelligence valuable. Users work in a natural, question and answer mode using familiar tools and processes. With minimal training, business users will quickly be using the system to put money into the bottom line. In addition to providing powerful and easy to use analytics, the Cirista products also provide the power needed for the more difficult BI scenarios. There are two conditions that make a BI system difficult to implement and slow to respond and consequently, not as valuable as you would want it to be. These are where there are many detail level records and/or where there are many attributes in the data to be analyzed. For example, a mid-sized grocer may generate 20,000+ checkout scan records per day. In a recent test project, we wanted to analyze data from 9 stores over a period of about six weeks resulting in 11 million detail level records plus a typical assortment of product and other records to analyze. Although just a test, even this project would be a challenge for a conventional BI/DWH system. The fully implemented BI system, with more than 700 million scan records, would require an investment in the millions and it still would not provide the complete, detail level analysis provided by the Cirista system. At the opposite extreme, we implemented a BI system for clinical trial analysis where there were fewer records but over 1,000 separate (non redundant) attributes involved in the analyses. Any specific analysis would involve an assortment of these fields and always in an unpredictable mix. This scenario would be impossible with any analytical tool that did not provide detail level access to every field of every record. However, this is the standard Cirista environment and the project was no more difficult than the simplest systems. In each of these cases, we went from receiving the data to delivering actionable business intelligence in less than one week and with less than 20 total hours of work. This included loading the data, reviewing it for quality, fixing a few data issues, performing ad-hoc analysis, generating dynamic analysis screens and then using those screens to find and display actionable results for business people in the form of Dynamic Drill Reports. In both cases every field of every record was available for use in the analysis In a typical business environment where the products are to be installed and used, the work we did on these projects would be similar to the work needed to implement the BI system and make it available directly to business users. The system would be delivering actionable business intelligence. Of course, there would be more time spent on preparing the production data transfer and load processes, user training and other startup activities. But even in a worst-case scenario, that included training, transfer and load and other startup activities, the Cirista solution will begin to deliver results as soon data is loaded into the system and should be fully functional in six weeks.

10 Cirista White Paper 10 Conclusion The SMB cannot approach BI in the same manner as a Fortune 1000 company does. They have neither the IT staff nor the funds that a big company has to support an on going traditional BI initiative. More importantly, they need to be more capable in their use of data to compete. With the Cirista approach, the small to mid size business can employ a powerful BI solution without making large investments in software and more importantly IT staffing. With the analytical power of the Cirista system, very difficult analytical challenges like one-to-one marketing are doable within the budget constraints that are a fact of life in the small to mid sized business.