TDWI strives to provide course books that are content-rich and that serve as useful reference documents after a class has ended.

Transcription

1 Previews of TDWI course books are provided as an opportunity to see the quality of our material and help you to select the courses that best fit your needs. The previews can not be printed. TDWI strives to provide course books that are content-rich and that serve as useful reference documents after a class has ended. This preview shows selected pages that are representative of the entire course book. The pages shown are not consecutive. The page numbers as they appear in the actual course material are shown at the bottom of each page. All table-of-contents pages are included to illustrate all of the topics covered by a course.

2

3 All rights reserved. No part of this document may be reproduced in any form, or by any means, without written permission from The Data Warehousing Institute. ii The Data Warehousing Institute

4 TABLE OF CONTENTS Module 1 Introduction to Business Intelligence Module 2 Business Application Fundamentals Module 3 Data Warehousing in a BI Program Module 4 BI Infrastructure Module 5 Summary and Conclusions Appendix A Additional Reading... A-1 Appendix B Bibliography and References B-1 The Data Warehousing Institute iii

5 Introduction to Business Intelligence Module 1 Introduction to Business Intelligence Topic Page Definitions and Concepts 1-2 A Framework for Business Intelligence 1-5 The Data Warehousing Institute 1-1

6 Introduction to Business Intelligence TDWI Business Intelligence Executive Briefing Definitions and Concepts BI Defined BUSINESS INTELLIGENCE Howard Dresner (Gartner Group) first defined business intelligence as shown on the right. As BI became a mainstream term, additional definitions have emerged, among them: BI is neither a product nor a system. It is an architecture and a collection of integrated operational as well as decision support applications and databases that provide the business community easy access to business data. (Larissa T. Moss and Shaku Arte, Business Intelligence Roadmap, Pearson Education, 2003) Business Intelligence a set of concepts and methodologies to improve decision making in business through use of facts and fact-based systems Howard Dresner, The Gartner Group The processes, technologies and tools needed to turn data into The goal is improved decision making. information, information into Concepts & methodologies Not just technology. knowledge, and knowledge into plans Facts & fact-based systems are necessary to implement. that drive profitable business actions. Business intelligence encompasses data warehousing, business analytic tools and content knowledge management. (David Loshin, Business Intelligence: The Savvy Manager s Guide, Addison Wesley, 2003) Business Intelligence provides the ability to transform data into usable, actionable information for business purposes. BI requires: Collections of quality data and metadata important to the business The application of analytic tools, techniques and processes The knowledge and skills to use business analysis to identify/create business information The organizational skills and motivation to develop a BI program and apply the results back into the business The foundation that enables BI is the enterprise architecture business, data and technical. A well implemented data warehousing program provides much of that foundation. BUSINESS ANALYTICS Business analytics is the process of using information (a collection of facts) to enhance knowledge and apply that knowledge to help a business achieve its objectives. Analytic applications provide tools to facilitate the business analytics process. 1-2 The Data Warehousing Institute

7 Introduction to Business Intelligence A Framework for Business Intelligence BI Components Framework Business Requirements BI Architecture Data Warehousing Program Management Data Resource Administration Data Sources Data Acquisition, Cleansing, & Integration Data Stores Information Services Information Delivery Business Analytics BI & DW Operations Development Business Applications Business Value STRUCTURE FOR BI COMPONENTS The BI components framework identifies the parts of a BI program and the relationships among them. The framework consists of three layers: The Business Layer includes those components needed for BI to fit seamlessly into business organizations, processes, and activities. Business Requirements, business valuation, program management, and development processes are the components in the business layer. The Administration and Operation Layer provides connections between business components and technical components. Components of this layer include BI architecture, business applications, data resource administration, and operations processes and services for BI and data warehousing. The Implementation Layer comprises all technical components needed to capture data, turn data into information, and deliver that information to the business. Implementation components are divided into two groups data warehousing and information services. Data warehousing provides the raw material from which information is produced. Information services extend beyond simply producing information to ensure that it is business aligned, valuable, and actionable. The Data Warehousing Institute 1-5

8 Business Application Fundamentals Module 2 Business Application Fundamentals Topic Page Business Requirements for BI 2-2 Business Value 2-3 Business Applications 2-5 Business Analytics 2-6 The Data Warehousing Institute 2-1

9 Business Application Fundamentals Business Value Business Valuation Models THE BUSINESS CASE The minimum business case for BI demands: (1) evidence that benefits are achievable, showing that investment will create a positive return), (2) a foundation to assess results and measure their value, and (3) a means to quantify and allocate costs. Both cost and value can be elusive. Some aspects of each are readily measurable. Others indirect costs and intangible benefits may be difficult to identify, and once known may be challenging to quantify. Cost and value assessments need to be performed at the start of a BI program, and continuously throughout the life of that program. Valuation models and metrics include: Return on Investment (ROI) A comparison of value received (revenue generated, costs avoided) to all costs incurred within a designated time period. ROI measures for BI programs include many challenges first in identifying the impact of information, then in determining the value of those impacts. Return on Assets (ROA) ROA complements ROI for valuation of a BI program. Where ROI measures value relative to expenses, ROI compares value received from BI (revenue generated, costs and risks avoided) with the value of assets essential to BI deployment. Assets to be considered go beyond the obvious technical infrastructure hardware, software, storage, etc. to include the data used to provide BI information. Many companies have tremendous, but dreadfully underutilized, operational data assets. BI offers opportunities to increase the value realized from data assets. Business Drivers forces that cause need to act Business Goals desired outcomes of actions Business Strategies planned means to achieve goals Business Tactics actions to implement strategies Business Results outcomes of business actions Return on Investment Return on Asset Total Cost of Ownership Total Value of Ownership value Outcome achievement, discovery Action insight, resolve, decision, innovation Knowledge recall, experience, instinct, beliefs Information facts, metrics Data cost descriptive, quantitative, qualitative Total Cost of Ownership (TCO) TCO models provide a structure to capture all costs associated with a BI program. Cost categories include hardware, software, staffing, and services. All costs including development, deployment, operation, support, and enhancement are included. TCO is total cost of the program from inception to the present. Measuring TCO for multiple time periods (program-to-date, year-to-date, current fiscal period, etc.) provides complete and substantial cost measures to be used in ROI and ROA calculations. Total Value of Ownership (TVO) TVO models provide a structure to capture value of all benefits derived from a BI program. Value categories include revenue received, costs avoided, and cost of risks avoided. All Measuring TVO for multiple time periods (matching those for TCO measures.) provides complete and substantial value measures to be used in ROI and ROA calculations. The Data Warehousing Institute 2-3

10 Data Warehousing in a BI Program Module 3 Data Warehousing in a BI Program Topic Page Warehousing Definitions 3-2 Warehousing Data Stores 3-3 Data Warehousing for Business Intelligence 3-4 From Data Warehousing to Business Intelligence 3-6 The Data Warehousing Institute 3-1

11 This page intentionally left blank.

12 Data Warehousing in a BI Program TDWI Business Intelligence Executive Briefing Data Warehousing for Business Intelligence From Data to Information DATA AND INFORMATION Data and information both have roles in BI, but they are not the ultimate deliverables of BI. Data access and information delivery both traditional data warehousing components have meaningful roles but limited utility in producing BI value. Data Access - Data access processes are those activities performed by business people who want to receive data that they will analyze, interpret, or use locally and individually. These processes are usually performed by people who have some skill in working directly with data. They are generally not strategic processes, but tactical or operational in nature. Data access processes are supported by tools with ad hoc query, report generation, and data retrieval capabilities. A single data access process may use several of these features. Data access capabilities support business processes and are most commonly analytical or operational in nature. Queries Downloads Reports (publish/subscribe) OLAP (slice, drill, etc.) Scorecards Dashboards Analytic Applications Information Information facts, metrics delivery Source Data Data Capture Data Transformation Data Cleansing Loading Data Warehouse Populating Data Marts Information Delivery Information delivery Data access processes differ from data access processes in two Data Marts very significant ways: (1) They provide information not just data; and (2) They are initiated by automated systems not by individuals. Further, information delivery processes are likely to participate in strategic and tactical activities, where data access processes work in the range of tactical to operational. Information delivery technologies include OLAP, which delivers metric information for interactive analytical, report publish & subscribe capabilities, dashboards and scorecards that present performance indicators, business metrics, alerts, trends, and forecasts in visual formats, analytic applications that package many information delivery capabilities for specific business purposes. Information delivery processes enable business activities such as fraud detection, supply chain optimization, performance management, etc. 3-4 The Data Warehousing Institute

13 Data Warehousing in a BI Program Data Warehousing for Business Intelligence From Information to Impact KNOWLEDGE, ACTION, AND OUTCOMES Awareness Insight Discovery Understanding Attention Determination Decision Implementation Tactical Effectiveness Strategic Position Outcome Source Data Data Capture Data Transformation Data Cleansing The true potential of BI is realized through business results the outcomes of applied information. Computer systems can deliver data and information, but people must apply that information in positive ways to receive maximum value. Information is most valuable when it has natural correspondence with the knowledge of those who will use it, enables people to take informed action, and helps to produce positive business outcomes. Data mining and applied analytics are the most common ways that information value is realized in a BI environment. Data mining is used to expand the body of knowledge. Applied analytics drive action and outcomes. Data Mining - Data mining is a process of Information Loading Data Warehouse Action knowledge discovery a means for finding facts, metrics new intelligence from collections of data. Populating Data Marts the process of discovering patterns that lead Knowledge Data to actionable knowledge from large data sets through one or more traditional data mining techniques, such as market basket analysis, Information Information facts, metrics delivery or clustering. (David Loshin, Business Intelligence: The Savvy Manager s Guide, Data Addison Wesley, 2003) access Data Marts Data mining is a process performed by business people, although it may be enabled by technology and assisted by IT staff. Fundamental to discovery of new knowledge is the ability to recognize discoveries in context of the business situations where they provide value and can drive effective business actions. While data mining may be goal-oriented (i.e., seek reasons why sales may be declining) or less specific (display the patterns inherent in a collection of data), the knowledge being sought is always that which is not yet known to the business person. Technology is an essential part of data mining; without it, people simply do not have the capacity to examine large collections of data and uncover implicit patterns. Data mining software applies logic and algorithms to inductively identify patterns and relationships inherent in data, and sometimes to infer rules from those patterns. Data mining enables discovery and facilitates awareness. People then interpret the patterns and inferences, and apply judgment to achieve insight and understanding. Applied Analytics - BI technology can provide analytics (trends, alerts, forecasts, and other business metrics) but technology can t act upon those metrics. Analytics are applied in context of business processes (sales, marketing, delivery, product development, etc.) and functions (managing customer relations, managing the supply chain, etc.). Like data mining, application of analytics is a process performed by business people. Awareness of trends and predictions, determination of responses, and decisions about the form and implementation of those responses are all part of applied analytics. Ultimately, the value of analytics is in the actions that result from them, not in the metrics themselves. Only by acting upon the information provided does BI provide value, increase effectiveness of business tactics, and enhance the strategic position of an enterprise. The Data Warehousing Institute 3-5

14 BI Infrastructure Module 4 BI Infrastructure Topic Page BI Infrastructure Components 4-2 BI Technology 4-3 BI Processes Program Management 4-8 Change Management 4-11 Data Governance 4-14 Data Warehouse Administration 4-16 Metadata Management 4-18 BI Roles and Responsibilities 4-21 The Data Warehousing Institute 4-1

15 BI Infrastructure BI Technology Kinds of Technology ROLES AND USES OF TECHNOLOGY BI and DW involve many kinds of technology. The technology framework shown below describes the kinds of technology at a high level. The technologies are further classified, as seen in TDWI s marketplace ( as tools for data warehouse analysis & design, data integration. Administration and operations, business analysis, analytic applications & application development, information delivery, and infrastructure and enabling technology. Each of these categories is discussed in greater detail on the following pages. The TDWI marketplace includes a directory of products in each category. Infrastructure B2E Portal (intranet) web desktop wireless voice Analytic Apps Development Data Access & Analysis Data Warehouse / Marts Analytic Applications Data Integration Data Resources Collaboration Text Analysis Content Management Data Warehouse Analysis & Design Data Integration Administration & Operations Business Analysis Analytic Applications & Application Development Information Delivery Infrastructure & Enabling Technology The Data Warehousing Institute 4-3 B2B & B2C Portals (internet/extranet) DATA WAREHOUSE ANALYSIS & DESIGN Designing a data warehouse involves creating, manipulating, and mapping models - conceptual, logical, and physical (data) representations of the business and end-user information needs. Some models are implied in source systems and must be reverse engineered, while others (such as those defining the data warehouse) must be created from understanding of requirements. Data warehouse designers map data between source and target models and capture the details of the transformation as metadata. Tools that support these various modeling, mapping, and documentation activities are known as data warehouse design tools. These technologies include: Data Modeling and Analysis - Individual tools as well as integrated tool suites are available to data warehouse designers. Modeling and analysis tool functionality includes applications for logical and physical database design, process modeling, DW mapping and transformation, visual data and data flow modeling, automatic code generation, data structure generation, support for reverse engineering, analysis for relational database schemas, meta data repositories, modeling templates, built-in de-normalization models and more. Warehousing Toolsets - Toolsets are available to support all tasks in all phases of data warehousing design, development, deployment, management, and information delivery. Various toolsets offer: o ETL, data cleansing, data aggregation, data flow engines. o Data modeling, design and prototyping, OLAP development. o Deployment & maintenance solutions, code and documentation generators, metadata management, usage monitoring. o Portals, dashboards, query tools and reporting.

16 BI Infrastructure TDWI Business Intelligence Executive Briefing BI Processes Program Management - Aligning Multiple Projects and Activities MANAGING FOR BUSINESS RESULTS Program management encompasses the disciplines and activities necessary to coordinate multiple simultaneous, overlapping, and interdependent efforts. Program management addresses both project efforts such as building a data mart or deploying an analytic tool, and ongoing efforts such as data warehouse operations or training and support. A BI program includes a governance structure and guiding principles, delineated architecture and methodologies, established standards, quality expectations, and measures of cost and value. The program s primary focus is on the strategic, long-term goals building an information infrastructure that will serve current business needs and evolve with the changing needs. Program (not project) focused -- A BI program is a large and complex undertaking with a broad strategic perspective and long-term focus. It is the structure that ensures cohesion among multiple projects. The program has long duration (for the life of BI within the enterprise) and measures success by business impact and ROI. Projects are one-time, time boxed activities with a specific business and/or technical objectives. Projects have start and finish dates, and are chartered to produce explicit deliverables. Projects are relatively short in duration, and measure success by on-time, within-budget delivery of products. Alignment of people, projects, and priorities The key to successful program management is alignment of people, projects, and priorities to do the best possible job of fulfilling the program s mission, enabling business strategies and tactics, and creating business value. Program management must attend to each of: Business priorities Satisfying needs based on relative business importance, and adapting when priorities change. Mission and goals Program mission and goals are stated and designed to support business mission and goals. Strategies and risks Risks are known and mitigated by active business participation and incremental projects. Multiple projects BI is deployed through many small projects. Each project has specific, defined deliverables Dependencies Relationships among projects, roles, and teams are known and actively managed. Cost and value Cost and value are regularly measured and program ROI is published and communicated. Business rules Business rules are captured, documented, and communicated. Rule changes are tracked. Infrastructure Scalable technical and non-technical infrastructure is actively managed and evolved. 4-8 The Data Warehousing Institute

17 BI Infrastructure TDWI Business Intelligence Executive Briefing BI Processes Data Governance INFORMATION ASSETS Data Governance is an ongoing process to manage and exercise control over business information assets a critical and valuable corporate resource. Effective data governance aims to improve and maintain data quality and consistency, reduce data duplication, formalize data accountability, align data assets with business rules and ultimately enhance business and IT effectiveness. A Data Governance Program formalizes business and technical goals - defining roles and responsibilities, assigning accountability, collecting of governance meta data, defining data policies, aligning relationships between data and associated business rules. Three primary roles data owner, data steward and data custodian need to be identified, and the people who fill those roles need to be committed and active participants in the data governance program. Governance and Business Rules - A business rule is a declarative statement of business logic, containing conditions or consequences that define how a business is run. Well defined and documented business rules provide a formal foundation for understanding the business, describe or guide policy or operational behavior and relate different business concepts together in meaningful ways. Business rules express user expectations about the data. Good data governance policies ensure that information assets are aligned with business rules. Governance and Information Quality - Data is the raw material from which information is built. Poor data quality leads directly to poor information quality; and poor information quality leads to inaccurate and unreliable scorecards, dashboards, and analytic applications. These, in turn, may lead to questionable business decisions. A good data governance program is attentive to data quality as one of its highest priorities. The program will provide ways to continuously improve the data across the organization by: determining the data quality expectations of the business, identifying metrics to assess the levels of data quality, identifying opportunities for improvement, eliminating problems at the source, and measuring continuous improvement against a baseline The Data Warehousing Institute

18 BI Infrastructure TDWI Business Intelligence Executive Briefing BI Processes Data Warehouse Administration Source Data Data Capture Data Transformation Data Cleansing Loading the Data Warehouse Populating Data Marts GETTING DATA TO THE WAREHOUSE Scheduling, Execution, and Monitoring - The first and most frequent responsibility in data warehouse operation is that of regular data refresh. It is essential to get new data into the warehouse and marts on a timely basis and as expected by the business. This responsibility consists primarily of scheduling and executing ETL processes. Knowing the state of source systems and data (processing errors, database backouts, schedule changes, etc.) is essential. Sometimes it is necessary to reflect operational system transactions in warehousing data at or near real time. This requirement most frequently occurs for the ODS, but real time data warehousing expectations are becoming increasingly common. These processes are not traditional ETL (extract, transform, and load) but are more likely to use a sequence of replicate, transform, and update. Synchronization with source systems when errors occur or database transactions are reversed is especially critical to real time refresh processing. Refresh Verification and Data Verification - Whether traditional ETL or real time refresh, the results of warehouse population processes must be verified. Check out results of every refresh process to ensure successful completion. Don t make the business discover & report errors that result from refresh Data Marts failures. Verification of data completeness and correctness helps to assure quality of the data in the warehouse and the information that will be derived from that data. Depending on the nature of the data, verification techniques may include row counts, control totals, sample set audits, balancing to sources, etc. Technical Support and Communication - Warehousing processes demand resources to support them in several circumstances: Scheduling typically depends on source system schedules. When exceptions to operational schedules occur, expect a domino-like effect in warehousing processes. When errors occur communication comes first, the focus on problem identification and correction Source system changes will drive maintenance changes to refresh processes and may require metadata to be modified. MANAGING TECHNOLOGY Backup and Restore - Platform management includes responsibility to prevent loss of data and provide points from which errorrecovery and be performed. Ability to backup and restore data and metadata are essential components of warehouse operation. Database Administration - Database management systems and the databases that use them require periodic maintenance and support. Managing data space utilization, balancing database workloads, and applying product updates are some of the common administration activities. Systems & Network Administration - Server systems and networks require routine maintenance and support, as well as troubleshooting and repair when problems occur. Systems and networks are made up of hardware and software that must be serviced like any other system. Configuration, workload balancing, problem detection and repair, monitoring, and tuning are among the common tasks. Performance and Use Monitoring - Users and uses of the data warehouse will change over time, and will fluctuate as the business experiences peak periods of activity. Monitoring performance and usage detects capacity problems early and provides information needed by database, system, and network administrators to optimize the environment for current usage patterns The Data Warehousing Institute

19 BI Infrastructure BI Processes Data Warehouse Administration Technology Administration - Technology administration attends to the ever-changing set of enabling technologies upon which a data warehouse depends. Current licenses, supported versions, and compatible product configurations are among the technology administration concerns. ENVIRONMENTAL SUPPORT Security Administration - Security services assure that warehousing data is accessed in accordance with security and privacy policies for the data. Administration responsibilities include processes and transactions to provide authorized data access and to terminate access privileges of persons no longer authorized. Active administration of security monitors authorizations against employee terminations, job transfers, etc. Growth Management & Capacity Planning - Growth is a certainty in the data warehouse. Increasing volumes of data, increasing numbers and kinds of users, expanding data storage requirements, and escalating network traffic are among the challenges of growth management and capacity planning. Monitoring growth, measuring capacities, and forecasting resource needs are among the responsibilities. Configuration Management - The data warehouse operates on a complex configuration of hardware and software platforms. Servers, storage systems, networks, databases, metadata tools, analytic tools, access technologies, and administrative tools must all work together seamlessly to provide reliable information services. Configuration management assures platform reliability in an environment of frequent technology change. Disaster Recovery - A successful data warehouse rapidly becomes a critical business information resource. Business continuity in the event of a disaster will include the need for information services. Informed decision-making, in fact, becomes a high priority. Interruption of decision-support information at a time that demands business-critical decisions is intolerable. The data warehouse and information services need to be integrated into the overall disaster recovery and business resumption plans for the enterprise. TRAINING AND SUPPORT Support Services - Support services may be as unique and varied as are warehouse users and their information needs. These services aid the business person to get value from the data warehouse, covering the entire range of assistance needs from determining requirements to understanding and applying information received. Support services are the core of customer care for BI and DW customers; their purpose is to make each customer experience with the warehouse a success. While they may take advantage of help desk and training services described below, support services cover a much broader range of assistance helping to determine what information is needed to respond to a particular business situation, helping to use tool to access data and produce a desired report, helping to publish a report to the company s intranet, putting a user-developed report in production to be generated routinely. These and many other examples illustrate the range of support services. The customer experience begins with a need for information and ends when information services result in business impact. BI/DW customer care attends to the entire customer experience. Help Desk Services - Help desk services respond to the set of recurring support needs where communication is the primary objective. Receiving inquires of all types and directing them to the right place for response is a primary help desk responsibility. Communicating solutions to common problems, informing the customer community of schedules and outages, and communicating technology and content changes may all be help desk services. Training - Training services respond to the set of reoccurring support needs where learning is the primary objective. Training topics range from skills to use DW and BI tools to understanding data warehouse contents. Training is an ongoing activity to keep knowledge and skills current as changes occur in the workforce, warehousing tools and technology, data warehouse contents, and information services. The Data Warehousing Institute 4-17

20 BI Infrastructure TDWI Business Intelligence Executive Briefing BI Processes Metadata Management KINDS OF METADATA Metadata is data that is used to develop, deploy, manage, and operate the BI environment. It is useful to think of metadata in four categories: Business Metadata - Business metadata describes the business meaning of BI and warehousing data. It captures information such as: business definitions, structure and hierarchy of the data, aggregation rules, ownership characteristics, subject areas, business-rule-based descriptions of transformation rules, definitions of business metrics, etc. Process Metadata - Process metadata describes the where (origins of the data system of record, database, etc.), the when (data of capture, frequency of capture, history of extracts and loads, etc.) and the how (tools used transformations applied, etc.) of BI/DW data capture, data transformation, and database loading. Technical Metadata - Technical metadata describes the physical locations, formats, and data types of data elements, file and table structures, database index schemes, implementation of data transformation rules, etc. Application Metadata - Application metadata describes how the data is accessed and used. It can describe when data is accessed, how frequently, who is using the data and how is it being used. SOURCES OF METADATA Metadata is found in a variety of sources including tools, application frequency of access? when accessed? systems, documents and business personnel. how accessed? etc. ETL and data quality tools can provide data transformation rules, load statistics, data lineage, program dependencies, etc. ETL execution can capture process metadata. Modeling tools provide logical and physical models and entity and attribute definitions. Analytic tools are the primary source of application metadata. Vendor applications may include data dictionaries and logical & physical models Business personnel & documentation are the best source for business policies and rules, ownership and stewardship of data, definitions of business terms and metrics, etc. USES OF METADATA Business Metadata data definitions metrics definitions subject models data models business rules data rules data owners/stewards etc. Technical Metadata data locations data formats technical names data sizes data types indexing data structures etc. Process Metadata source/target maps transformation rules data cleansing rules extract audit trail transform audit trail load audit trail data quality audit etc. Application Metadata data access history: who is accessing? Typical business users need metadata to understand data warehouse contents, and determine the meaning and source of information that they see in reports, queries, and analytic applications. Technical users need metadata to maintain and extend the warehouse, perform impact analysis and administer such activities as archiving or performance tuning The Data Warehousing Institute

21 BI Infrastructure BI Processes Metadata Management METADATA CHALLENGES Metadata management is a difficult and challenging aspect of business intelligence. Metadata dependencies are found throughout BI and DW. Every technology and every activity has some need to access and use metadata, and many of them capture and store metadata. Among the metadata challenges: Metadata Requirements - Understanding metadata requirements is fundamental to formulating a metadata strategy. Requirements need to consider (1) Why metadata is needed, (2) How much and which metadata is needed, and (3) How much metadata integration is necessary. The successful metadata program identifies essential metadata based upon business needs, selects a manageable initial scope (& cost) with clearly identified business benefits and then incrementally rolls out metadata capabilities and achieves associated benefits. Metadata Collection - Gathering all of the metadata is a difficult activity. Identifying the sources and capture points is challenging enough. Once they are known, the metadata collection tasks may be labor intensive. These tasks include Collect and store technical metadata about warehousing data. Build process metadata capture capabilities into ETL. Author business metadata about warehousing data. Find and/or create business & technical metadata about sources. Metadata Integration - Multiple tools capture different and sometimes overlapping metadata. Integration involves identifying all the tools, applications, and processes that capture metadata; knowing what metadata they capture; and determining how (in a sustainable and cost effective manner) to realize integration across those components. Metadata Maintenance - Keeping metadata complete and current is a continuous and resource intensive effort. The volume of metadata will increase as business users discover new ways to use information and new business questions are asked. Careful attention to standard processes and effective application of available tools will help keep the job manageable. Metadata Access - Metadata access for the business is ideally integrated into the analytic tools being used. Metadata provides context for information, and is most effective in doing so when metadata access is seamless. Providing access for all metadata users business and technical is challenged when metadata storage is not integrated but an integrated access facility is desirable. METADATA STRATEGIES The biggest challenges in metadata management are those of collection and cohesion of metadata captured independently by multiple tools and technologies. The most effective strategy is the one that cost-effectively meets current and near-term needs with ability adapt as those needs grow and change. Metadata strategies include: Integrated Tools Suite - Single vendor or vendor partnering solutions may provide a strong metadata component fostering integration across multiple metadata types and including metadata capabilities in the development and access functions of their tools. This approach can help mitigate challenges of integration and access and provide for automated collection of process and technical metadata. This approach assumes a vendor solution that is a good match with requirements. Build a Repository - Building your own metadata repository creates a single, integrated store of all metadata needed for the BI environment. Given specific and unique metadata requirements, this approach can provide strong integration in a multiple tool & technology environment. However, building a repository is a complex, resource intensive and time-consuming effort. In addition to building the repository, you will need to develop and maintain the processes that populate the repository, such as multiple interfaces to tools and applications. The Data Warehousing Institute 4-19

22 BI Infrastructure BI Roles and Responsibilities Who Does the Work? MANY ROLES IN BI Developing, operating, supporting, and sustaining a BI program involves many roles and responsibilities. Both business people and IT staff have crucial roles that the fill to ensure a successful and valuable program. These pages discuss BI roles classified into the following categories: Business Roles Management Roles Architecture Roles Requirements Analysis Roles Design & Construction Roles Infrastructure Roles business rules specification BI Program sponsorship program management data governance BI Projects integration design database development ETL development BI Operations project management data integration & cleansing data acce3ss, analysis, & mining business metrics usage system & database administration process execution & monitoring training & support data mart development BI application development source data analysis business requirements definition BUSINESS ROLES metadata management architecture specification quality management Business roles are business unit responsibilities that should clearly not be assumed by or delegated to an IT organization. Simply understanding the needs and processes of the business is not sufficient qualification to fill these roles. They re about ownership. They demand a stake in the game. Thus, even an exceptionally business-savvy IT person is likely to be under-qualified for the job. Business roles include: Sponsor Business Subject Expert Knowledge Worker Data Owner Data Steward Program Manager Project Manager Information Architect Acquisition Architect Technical Architect Business Requirements Analyst Source Data Analyst ETL Developer Database Developer Front-end/OLAP Developer Technology Specialist Systems Administrator Database Administrator Metadata Administrator Quality Administrator Customer Service & Support Specialist Trainer Sponsor - Sponsors establish the charter and high-level goals, acquire funding, provide political will, and secure resources for BI/DW programs and projects. Sponsors also have responsibility as the ultimate resolution place for issues and conflicts. Program sponsors have interest in program, project, and operation kinds of work, and direct responsibility for program-level work. Project sponsors have responsibilities associated with the specific projects that they sponsor, and have interest in the operation and support implications of those projects. Effective sponsors are in a position to have influence across the entire scope of business to be affected by the programs and projects that they sponsor. Business Subject Expert - Subject experts are responsible to provide topic knowledge to a project within a specific domain. This is a project role, and is the primary way that projects acquire business knowledge. Expertise about business data, rules, processes, organizations, systems, and needs are all subject expert responsibilities. Subject experts need to have good conceptual knowledge of their subject domain and sound practical experience working within that domain. They also need the skills to communicate their knowledge to other project team members. End User - End users are responsible to get value from information resources. This is an operational responsibility to understand BI/DW concepts and environment, understand the kinds of information available, access information and apply it to achieve business impact. To fulfill these responsibilities, it is necessary to acquire knowledge of and skill using data access and analytic tools. The end The Data Warehousing Institute 4-21

23 BI Infrastructure TDWI Business Intelligence Executive Briefing BI Roles and Responsibilities BI Organizations Business Organizations Sponsor Business Subject Expert Knowledge Worker Data Owner Data Steward Program Manager Project Manager Information Architect Acquisition Architect Technical Architect Business Requirements Analyst Source Data Analyst ETL Developer Database Developer Front-end/OLAP Developer Technology Specialist Systems Administrator Data Administrator Metadata Administrator Quality Administrator Customer Service & Support Specialist Trainer BARRIERS IT Organizations Sponsor Business Subject Expert Knowledge Worker Data Owner Data Steward Program Manager Project Manager Information Architect Acquisition Architect Technical Architect Business Requirements Analyst Source Data Analyst ETL Developer Database Developer Front-end/OLAP Developer Technology Specialist Database Administrator Systems Administrator Metadata Administrator Quality Administrator Customer Service & Support Specialist Trainer TRADITIONAL ORGANIZATIONS Traditional organization structures for information technology initiatives divide roles between those held by IT organizations and those held by business organization. With two distinctly different types of organizations needing to work together for success, a set of business rules formally or informally describe the working relationships. Common models for the business relationships include contracts, partnerships, and collaboration. Each is summarized in the chart below with respect to six universal building blocks of working relationships. CONTRACT PARTNERSHIP COLLABORATION FOCUS Process-centric Organization-centric People-centric BUY-IN Negotiation Consensus Trust & commitment CONTINUITY Document and sign Discuss and document Strong leadership CULTURE Us-and-them Defined roles and responsibilities Key individuals CHANGE Limited responsiveness Adjust the agreement Adapt as needed TEAMWORK Inhibited Facilitated Demanded 4-26 The Data Warehousing Institute

24 Summary and Conclusions Module 5 Summary and Conclusions Topic Page BI Best Practices 6-2 References and Resources 6-3 The Data Warehousing Institute 5-1

25 Summary and Conclusions TDWI Business Intelligence Executive Briefing BI Best Practices Positioning for Success Successful BI Programs will: LESSONS LEARNED 1. Understand the Drivers 2. Measure Results 3. Make it a Business Initiative 4. Practice user first Design 5. Create New Value 6. Attend to Human Impacts 7. Focus on Information and Analytics 8. Practice Active Data Stewardship 9. Manage BI as a Long-Term Investment 10. Reach Out with BI/DW Solutions This list of best practices is adapted from an article TDWI s FlashPoint e-newsletter of April 10, Each practice is briefly described below. A reprint of the entire article is reprinted in Appendix A. FlashPoint is distributed regularly to TDWI members. Each issue contains thought-provoking articles from leading BI/DW consultants and practitioners: Strategic Positioning - BI success begins with clear understanding of the reasons that BI is important to your business. Knowledge of both business drivers and BI program drivers defines the strategic and tactical positioning of a BI program. Measured Results - Applied analytics succeed in organizations that use metrics effectively. When you acknowledge the BI/DW program as both a business process and an investment, it becomes an obvious target of applied metrics. Establish measurable goals and define actionable metrics for the program. Business Initiative - Successful BI engages the business. It is a business first, technology second endeavor. Technology can support BI, but it can t create BI. Technology organizations can deploy BI solutions, but they can t create BI cultures. User First Design - BI/DW products only deliver value when they are used by people conducting business activities. Design first for the user experience. Start with what the user will see and how they will interact with BI products. Give particular attention to how BI solutions will fit seamlessly into day-to-day business activities. New Value - BI programs are costly initiatives, justified solely by the business value that they return. Sufficient value isn t likely to be obtained simply through new ways to deliver old information. Seek opportunities for and strive to deliver value-added information. Don t just integrate data enrich information. Human Impacts - BI success and effectiveness depends on people. From those who define and articulate business strategies to those who employ BI products in their day-to-day work, BI changes jobs. It creates new roles and responsibilities, demands new skills, and changes the way that work is done. Business professionals and seasoned IT people will both experience significant change. Information & Analytics - Information is more valuable than data. Data is only the raw material that is used to create information. Position data warehousing as the supply chain for business information, and DW technology as under the hood stuff that is necessary to power business intelligence. Active Data Stewardship - If data is the raw material from which we create information; then reliability of information directly depends on the quality of the data that is used to produce it. Good information depends on good data; and good data is a product of good data management, not of good fortune. One of the keys to BI success is business-managed data & information resources, implemented through active and ongoing data stewardship. Long Term Investment - Strive for lasting value from the BI program. While quick-strike successes create political goodwill and heroes-for-a-day, a sustainable BI program is one that continuously provides meaningful information, useful analytics, and measurable impact. Reach Out - Extend the reach of BI as far as practical. Each business process touched by BI is a potential opportunity to receive value, as is each person who uses the BI products. 5-2 The Data Warehousing Institute

26 Additional Reading Appendix A Additional Reading Topic Page Total Cost of Ownership for BI and DW A-3 Ten Best Practices for Business Intelligence A-7 The Data Warehousing Institute A-1

27 Additional Reading Ten Best Practices in Business Intelligence and Data Warehousing (a reprint of the TDWI Flashpoint from April 10, 2003) As we move beyond data warehousing (DW) and into business intelligence (BI), the nature of BI programs and projects is changing. Advances in technology, increasing complexity, rising business expectations, and cost-consciousness are among the primary contributors to the change. With changing programs and projects, we also experience changes in best practices for BI/DW. DW best practices of recent years addressed topics like which data to put in the warehouse, how to organize and structure data, and support of end-user access and inquiry. Today s BI best practices attend more to business factors than to technical considerations, addressing issues like business value, effect on business processes, and impact on people and their jobs. This article summarizes some practices that I have observed as common among successful BI organizations. 1. Understand the Drivers BI success begins with clear understanding of the reasons that BI is important to your business. Knowledge of both business and BI program drivers defines the strategic and tactical positioning of a BI program. Business drivers are external forces that cause an enterprise to act and change the things to which business strategies must respond. Changing economy, changing workforce, and changing regulations are common business drivers. BI/DW program drivers establish a basis to align information technology with business strategies. Drivers like market penetration, performance improvement, cost reduction, process optimization, and risk mitigation establish the context in which information will be used and analytics will be applied. Understanding both business and program drivers is essential to realize value from a BI investment. 2. Measure Results Applied analytics succeed in organizations that use metrics effectively. When you acknowledge the BI/DW program as both a business process and an investment, it becomes an obvious target of applied metrics. Establish measurable goals and define actionable metrics for the program. Measure both the financial and cultural impacts of the program. Include total cost of ownership (TCO) and total value of ownership (TVO) as financial measures. They are necessary to derive metrics like ROI and NPV, and they provide measures needed to manage BI resources wisely and cost-effectively. Define cultural measures and metrics to manage how BI is used. Collect and apply data about who is using BI products, which products are most (and least) used, how they are used, and how they affect business activities. Cultural metrics allow you to extend the reach of BI, to recognize new needs and opportunities, to enhance valued BI products, and to retire unused products. 3. Make It a Business Initiative Successful BI engages the business. It is a business first, technology second endeavor. Technology can support BI, but it can t create BI. Technology organizations can deploy BI solutions, but they can t create BI cultures. Business engagement begins with business sponsorship (a frequently observed DW best practice) but extends well beyond sponsorship. The business must not only sponsor the BI program; it must own the BI products. With ownership comes responsibility to govern and be stewards of data, to integrate BI solutions into the fabric of daily business The Data Warehousing Institute A-7