Constructing Your Data Warehouse

Transcription

1 Constructing Your Data Warehouse

2 Experience, Expertise, BIAS Sr. Vice President for Finance & Information Systems VGH & UBC Hospital Foundation Principal and Founder SupportingAdvancement.Com SupportingFundraising.Com Finance, Information Systems, Operations (Most of Career) Large, Medium and Small Organizations Not for Profit and Private Sector Financial Statements, Audit, Investment Management Transaction Processing and Management Technical Infrastructure Database Management, Reporting and Data Warehousing Web, Online, Social Media, Marketing Part-Time Consulting

3 Lessons Learned Ongoing Support from Sponsors is Critical Planning is Not Always Fun But Essential Prioritize Since You Can t Do Everything Be Creative Change Management & Communication is Key Success Requires Changes in Jobs, Roles and Responsibilities Everything Takes a Lot of Time and Effort Every Organization has a Variety of Issues and Opportunities Try and Learn from Past Mistakes and Successes Metrics and Measurement Have Fun

4 The Information World

5 The Information World Rapidly Changing Increased Complexity More Visual Competition Cloud Mobile Big Data Privacy and Security Public Perception Information Ethics Information Neutrality

6 Increased Complexity

7 More Visual

8 The Information Consumers

9 The Information Consumers Desktop Mobile Search Apps Social

10 The Information Consumers Search Fast, Easy, Iterative Mobile Anyplace, Anywhere, Anytime Apps Pick the Best Software to do the Task Social Ask/Tell Others What Works/Is Is/Not Best Self Service Approach to Tasks/Knowledge How do we leverage our organizations to take advantage and promote self service? More challenging in small to mid-size shops?

11 The Information Consumers

12 The Information Consumers High Expectations, Limited Patience, Short Attention Better Trained, Quick Decisions Interested in Discovery, Asking About Outliers Expectations for Benchmarking

13 The Information Tools

14 The Information Tools Visual Analytics Statistics and Modeling Access to Public Data Sets In Memory Data Analysis Web Enabled Cloud Hosted Available Via Mobile Many Old Limitations are Behind Us: Disk Space, Bandwidth, CPU Processing, Memory, Etc.

15 The Information Tools Technology Business Practices User Experiences

16 Viz of the Day

17 Our Information Environments

18 Our Transactional Systems Work Well for Capturing Details We re Used to Working In Them They re Not Good at Providing Aggregate Information They Inhibit Discovery Long Cycle Time Between Upgrades Staff who tend to more often understand the complex relationships between data are for the most part working in gift and records processing and annual giving. Leadership and management are removed from data complexities

19 Our Reporting Environments Query Engines Imbedded in Transaction Systems are Inefficient in Providing Information Canned Reports in Transaction Systems are Generally not Always Useful List Based Cultures Lots of Rework of Data in Spreadsheets Stand Alone Reporting Tools With Complex Logic Imbedded in Reports Sharing Information Via Little Automation of Reporting Environments Limited Tools for Discovery Printed Reports Long Cycle Time for Report Development by Technical Staff

20 Logic Imbedded in Reports

21 Our Reporting Environments

22 Report Critique

23 Where now?

24 Where now? Significant Redesign of our Information Environments Learning and Deployment of New Tools Renewed Focus on Data Integrity and Quality Ability to Integrate Data from Multiple Sources New Skill Sets for Information Developers New Skill Sets for Information Consumers Business Technical Artisan Statistics Developers ********** ********** ***** ******* Consumers ********** ***** ******* *******

25 Training

26 We Shall Overcome

27 What is a data warehouse? Repository of an organization's electronically stored data. Data warehouses are designed to facilitate reporting and analysis. A data warehouse houses a standardized, consistent, clean and integrated form of data sourced from various operational systems in use in the organization, structured in a way to specifically address the reporting and analytic requirements. (Wikipedia)

28 How is a data warehouse created? Data from the transactional system is loaded into the data warehouse every night from a variety of sources. Data is reformatted in ways that facilitate analysis and reporting. (Extract, Transform, Load) Data cleanup and other statistical tables are updated. Transformed Tables Used for Reporting Transactional System Loading Reformatting Calculating Data Warehouse Tables

29 Data Warehouse Overview

30 Data Warehouse Overview

31 Data Warehouse Benefits One Version of The Truth. Business Logic is encapsulated in the programs used to create the data warehouse. If you need to change business logic you change it in one place and all information and reports based on this logic will change accordingly. Real Time. Any time you get information in less than real time, you need to significantly re-architect your information environment. Ralph Kimball. Information consumers don t have to wait for information and don t have to manipulate or modify it to get answers to their questions. Leverage. Reusable code and objects. Ability to create/combine/enhance your data in ways that are not possible in transactional and other systems. User empowerment. Reduction in time taken to produce actionable information.

32 One Version of the Truth What is a gift? What is an annual gift? What is a major gift? How do we calculate our total pipeline? How do we measure our data quality? How is a fundraiser s performance changing over time? What is a donor s cumulative lifetime giving? How can we predict what a prospect will do next? Are we collecting our pledges on time? How does our giving correlate with the stock market?

33 Real Time Financial Statements (2-3 Weeks) Recompiling Data into Spreadsheets (2-3 Days) Reports From Data Warehouse (Less than 24 Hours) Visual Analytic Tools (Can be Real Time) Data Mining Projects (Depends Up to a month?)

34 Real Time

35 Leverage Business Logic and Complex Calculations in One Place Rename Fields So They are More Descriptive Reduce Turnaround Time on Report Requests Much Easier to Share Consistent Information Easily Discover Data Quality Issues Data Warehouse Reinforces Data Quality Reduce User Training More Flexibility in Reports Combine Data from Multiple Sources Add New Data That s Not Captured Anywhere Else Empower Information Consumers Metrics to Track Usage and Success

36 Leverage

37 Basics for Any Project

38 Plan Organize Planning Process Conduct Research Construct Plan Formalize the Planning Process Implement Plan Evaluate Metrics and Outcomes Revise Plan

39 Document Your Plan What s in it for me?

40 Communicate What s in it for me?

41 Next Steps

42 Determine Your Information Goals Improve information required to manage organization performance. Provide an environment where information can be easily consumed and where information discovery is enabled. Reduce the amount of manual effort required to compile information by providing flexible automated reporting that is in a final format. Improve data quality. Establish an organization wide priority for information goals.

43 Determine Your Information Goals

44 Information Goals

45 Evaluate and Select Information Tools Integration With Existing Technology Learning Curve and Training for Staff Licensing Costs Technical Infrastructure Requirements Vulnerability to Knowledge of Specialized Tools Ability to Hire/Contract People With Requisite Skills

46 Evaluate and Select Information Tools Choose Your Analytics Tool First (Tableau, QlikView, Others ) Choose Your Standard Reporting Tool Second (Reporting Services, Crystal Reports, Others ) Other Tools Third Why choose this order?

47 Tableau Powerful and Flexible Relatively Easy to Use Large User Community With Lots of Resources Web Enabled Can Be Deployed by Mobile Not Always Intuitive or Easy to Use Not Designed for Printing Not Inexpensive Specialized Expertise User Needs Business Analyst Skills

48 Reporting Services

49 What is Reporting Services? Integral part of Microsoft s suite of business intelligence software products. Reporting tool which is similar to Crystal Reports and other reporting tools. Tool that provides reports to users via a web site and web browsers however you can subscribe to reports and have them automatically delivered via . Tool you can use to develop your own reports.

50 Why use Reporting Services? Powerful and flexible. Inexpensive licensing. Mainstream product. Learning curve is reasonable Integrated with Microsoft SQL Server and related suite of products. Technical advantages. Reporting tool for future versions of Raiser s Edge (integrated into CRM.)

51 Why Reporting Services

52 Excel Microsoft s User Facing BI Tool Powerful, Flexible, Easy to Use, Inexpensive Consider Integration with SQL Server and SharePoint Massive User Base PowerPivot, Data Mining, Power Query

53 Others Data Mining and Statistical Software (Contract Out or Develop In-House Skills) Query Engine in Fundraising System Dashboards in Fundraising System Canned Reports in Fundraising System Downloads, Reports and Information from Other Systems.

54 Data Warehouse Design Appeal Codes

55 Appeal Codes How did we ask for the gift? The appeal code attached to the gift determines which fundraising program get s credit for the gift. (Can be different from individual fundraiser credit.) Each gift can have just one appeal code. Examples: DMLAP13 Direct Mail July Lapsing Donors for 2013 DMWR13 Direct Mail Winter Renewal 2013

56 DMLAP13 July Lapsing

57 Back to the Analytical Report

58 Back to the Analytical Report Complex logic in report and challenging to fix if something went wrong or report needed modification Only one person could maintain extreme vulnerability to technical skills (organizational decision) Monthly Business Process 1. Close Gift Processing 2. Run Analytical Report 3. Distribute to Fundraising Programs 4. Run Queries to Verify Report 5. Reconcile to Financial Statements 2-3 weeks minimum time after month end for fundraising performance information.

59 Appeal Schema in the Transaction System

60 Appeal Schema in the Transaction System

61 Nightly in the Data Warehouse Extract data from the transactional system. Transform the data to facilitate reporting, analytics and data quality. Load the data into the data warehouse. Review and fix any job errors. Transactional System Loading Reformatting Calculating Data Warehouse Tables

62 Design Script to Create dw_appeals What do we want to add to the table to make it more useful for reporting and analytics? What dropdowns and descriptions will be used in the reports? Will users download the information directly into spreadsheets or access using other tools? How will the information display? What indexes will be required to help performance? What coding and documentation standards are to be used within the script? What are the quality control steps to determine if information in the table is accurate? How will this new table help us reduce our own amount of work?

63 Sample SQL Script for Appeals Table

64 Sample SQL Script for Appeals Table

65 Sample SQL Script for Appeals Table

66 Appeal Schema in the Data Warehouse

67 Value Add Renamed Columns Easier to Read Organized Logical Names Consistent Names Themed to Table Makes Joins Easier

68 Value Add New Columns Extend capabilities of transactional system with new information Calculations don t have to be done in reporting or analytic tools Saves having to make format changes 01 April Fiscal Month Sorting is imbedded with descriptions, no additional joins to code tables in reporting tools

69 Value Add Leverage Business Logic in One Place Reuse of Formatting Multiple Scripts Using Functions and Stored Procedures (Fiscal Year Description) Leverage Appeal Information in Multiple Places Columns for Grouping and Sorting Included (01 January) Null Values Populated (*** Unknown Fundraising Program***) New Calculations

70 Value Add Leverage dw_gifts (Gifts table in data warehouse.)

71 Begin Design Using Analytics Tool Ask Questions of the Data Iterate Through Answers with Users Look for and Understand Outliers Review Column Names Review Descriptions in Rows Review Populated Columns for Null Values Review Complexity of Joins Review Calculated Columns

72 Design Reporting Services Reports

73 Flexibility A personal cup venti sugar free vanilla non fat and soy ½ pump almond and a ½ pump mocha two pumps sugar free cinnamon dolce with upside down whip caramel macchiato

74 Flexibility

75 Real Time

76 User/Developer Empowerment

77 User/Developer Empowerment

78 Value Add Monitor Errors

79 Value Add Monitor Errors

80 Value Add Training

81 Value Add Analytics If we have time at this point, we ll show a real time demo using Tableau to help demonstrate some of the concepts we ve already talked about.

82 Don t Forget The Human Component Reports Help With Information Democratization Structured Requests for Pieces of Information Help Educate Users Self Service to Run or Subscribe to Reports Same Look and Feel for Reports Increases Familiarity with Understanding Underlying Information Orientation Sessions with All New Staff Training Sessions to Reinforce (Appeals, Data Warehouse, Data Cleanup)

83 Don t Forget The Human Component Integrate Analytics Into Meetings Offer to Project Information on Screen Attend Meetings to See How Information is Used Talk to Managers Whenever You See Spreadsheets Control Design Process for New Information Monitor Usage Wherever You Can Office Walkabouts and Observation Check Printers for Output

84 Appeal Request Form

85 Appeal Request Form

86 Appeal Request Form

87 Metrics of Success

88 Metrics of Success What s actually used in meetings now (no spreadsheets unless an emergency information need) Using information from data warehouse directly for Board, no intermediate steps Stories from Staff It used to take weeks, many queries, a ton of effort Eliminated the use of one off Crystal Reports More awareness and headway on data cleanup and quality Reduction in queries run in the fundraising application.

89 Ongoing/Future Challenges Users not always comfortable with asking for new reports or sets of information. In-house business analyst, statistician and similar skills still need to be enhanced for non-technical staff. Expertise in visual analytics tools currently limited to technical staff (data complexity and licensing costs.) Lots of printed reports. Sometimes too much faith in the accuracy of information in the new reports. Inflexible bureaucratic infrastructure for new projects.

90 Thanks for your Time and Attention

91 A Few Resources

92 CASE Advancement Services Books

93 CoolData Blog

94 Data Warehousing Toolkit

95 Data Warehousing Institute

96 Fundraising Analytics

97 International Institute of Business Analysts

98 Many Eyes

99 Microsoft Lots of online resources for all products, developer networks and local user groups. Communities of practice such as the SQL Server User Group.

100 Periodic Table of Visualizations

101 SupportingAdvancement.Com

102 SupportingFundraising.Com

103 Tableau Software

104 Questions?