Data Warehouse on a Budget How to really do more with less

Transcription

1 Data Warehouse on a Budget How to really do more with less Discussion October 11, 2009

2 Agenda Essentials (foundation products) Key components (design build products) Inexpensive, high quality substitutes A word about accelerators How to deliver value quickly Other options Alternatives to mainstream thought 2

3 About me Led design/build of seven (7) large scale (over 5TB) data warehouses designed, built, and deployed in the Financial Services, Transportation, Supply Chain, Retail, Utility, and Professional Services industries Over twenty-five (25) data marts (special purpose subject areas) designed, built, and deployed across a wide variety of industries Eight (8) data warehouse executive assessments prepared and delivered for management review and action. In addition, five (5) detailed business cases prepared to support the investment in the analytic environment Five (5) commercial off-the-shelf products developed and marketed worldwide to the Software Engineering and Healthcare industries 3

4 Essentials Foundation components 4

5 Essentials Fundamental Pattern 5

6 Essentials Full Deployment 6

7 Essentials - Federated Environments Claims Processing SAP Financials CRM 3rd Party E-Commerce Warranty Analysis AWARE Common Staging Area Federated Meta Data Repositories Real Time ODS Federated Claims Processing Data Marts Actuarial Data Warehouse Federated Financial Data Warehouse Federated Marketing Data Warehouse Real Time Data Mining and Analytics Subset Data Marts Real Time Segmentation, Classification,Qualification, Offerings, etc. Analytical Applications 7

8 Essentials Common Information Model 8

9 Essentials EII 9

10 Design Build Components 10

11 A closer look 11

12 Key Processes 12

13 Test Automation Strategy Test Autom ation Strategy Unit B asics Test Execution Recorded Test Scripted Test Data Driven Test Test Autom ation Fram ework Test Runner Test Case Object Test Autom ation Fram ework Test Fixture Strategy Fresh Fixture Shared Fixture Test Discovery Test Enum eration Test Selection Test D efinition Standard Fixture Im m utable P re-built Test Case Object Test Method Assertion Method Test Case Class Four Phase Test Assertion Message Fresh Fixture Patterns Inline Setup Delegated Setup Im plicit Setup Shared Fixture Patterns Construction Lazy Fixture Setup Suite Fixture Setup Decorated Setup Chained Test Creation Method Object Method Access Finder M ethod Fixture Registry Delta Assertion Result Verification Patterns Fixture Tear D ow n Patterns State Verification B ehavior Verification Inline Tear Down Im plicit Tear Down Expected Object G uard Assertion Custom Assertion Verification Method Delta Assertion Garbage Collected Tear D o w n Autom ated Tear Down 13

14 Test Automation Strategy - Realized 14

15 Data Quality Data Quality Process Data Profiling Measure Quantifies the number and types of defects Analyze Data Cleansing Parse Assess the nature and cause of the defects Standardize Isolate and identify data elements in data structures Normalize data values and formats according to business rules and third-party references Data Enhancement Enhance Match and Consolidate Match Consolidate Append additional data enhancing the information value Identify duplicate records within multiple tables, databases Combine unique data elements from matched records into a single source Correct Management Reporting and Oversight Verify, scrub, and appends data based upon algorithms, business rules provided from a secondary source Provide reporting within the data quality process Report 15

16 Data Quality Why it is needed SQL Server 2008 Data Profiling Task in Integration Services 16

17 Choices Enabling Technologies 17

18 The choices ORACLE, SAP, IBM, Informatica Powerful Expensive Demands high skill levels to deploy successfully Microsoft Good, well rounded general purpose platform Missing key management and meta-data elements Open Source (Pentaho, Jaspersoft, and Infobright) Validated the market for open source BI reporting and ETL tools Good, special purpose tools in the right hands (Talend) Alternatives Wherescape RED Special Purpose Tools (SeeWhy, Pervasive) 18

19 Total Cost of Ownership Labor intensive Subject to Vendor Driven Architecture (VDA) Expensive (maintenance, hidden support costs)? Missing critical management components Customization and development costs Meet organizational capability and align with objectives Expensive and time consuming if not JAVA or.net UNIX or Microsoft Technical debt Quick and dirty is expensive Should invest more heavily in design 19

20 How to save 10 million dollars Replace AIX with Linux Websphere with JBOSS Domino with Alfresco or Drupal (ECM) Cognos with Pentaho Tivoli Monitoring with Hyperic Tivoli Netview with Zenoss Tivoli (Netcool) with OpenNMS Tivoli Configuration Manager with Puppet Tivoli Provisioning Manager with OpenQRM while staring into the abyss John Willis: IT Management and Cloud Blog 20

21 Seriously Most of our costs are in our people (4-5x) Development Support Maintenance Need for consistent, repeatable process controls Enable cost efficiency Deliver information products faster and less expensive Reduced complexity Component reuse Improved communication Leverage standardization benefits Less variance in work products Solve problems once Improved quality (defects caught earlier in cycle) Adopt standardized reference models, and templates 21

22 Seriously Open Source may not be so Open Align with internal skills and core competencies UNIX vs. Windows Java vs..net Perl vs. Powershell or WSH PHP vs. ASP Windows DW Stack may not be complete Management Metadata Flexibility Do not try to build a system whose complexity exceeds the organization's capabilities to deliver 22

23 What is the best solution on a budget? Probably something in between Platform (don t forget virtualization in development) Database and Storage Architecture Middleware Data Profiling and Quality Tools Configuration Management and ALM Test Automation and Continuous Integration Cruise Control NANT MAVEN Reporting and Information Delivery Reporting Services Excel (Server based zero footprint) 23

24 Inexpensive, high quality substitutes Alternatives to mainstream thought 24

25 Zenoss Core - monitoring and systems management 25

26 Puppet Automated Systems Administration 26

27 Subversion Version Control 27

28 Maven and Eclipse Build and Manage Projects 28

29 Pentaho (BI-Suite) 29

30 Jaspersoft 30

31 Talend 31

32 INFOBright 32

33 Protégé and the Essential Architecture Project 33

34 DB Designer 4 34

35 A word about accelerators 35

36 Wherescape RED 36

37 Wherescape RED 37

38 Methodology Along the way 38

39 MIKE2.0 (Methodology) 39

40 Comprehensive Process Models 40

41 Self documenting 41

42 Questions and reference links Wherescape RED Talend Essential Project Mike Pentaho BI Enterprise Suite nfobright

43 Questions and reference links JasperSoft John Willis: IT Management and Cloud Blog Cruise Control Maven NANT Subversion, Puppet

44 Data Warehouse on a Budget How to really do more with less Thank You

45 Data Warehouse on a Budget How to really do more with less Contact: j.parnitzke@comcast.net Mr. Parnitzke is a hands-on technology executive, trusted partner, advisor, software publisher, and widely recognized database management and enterprise architecture thought leader. Over his career he has served in executive, technical, publisher (commercial software), and practice management roles across a wide range of industries. Now a highly sought after technology management advisor and hands-on practitioner his customers include many of the Fortune 500 as well as emerging businesses where he is known for taking complex challenges and solving for them across all levels of the customer s organization delivering distinctive value and lasting relationships. Blogs: Applied Enterprise Architecture (pragmaticarchitect.wordpress.com) The Corner Office (cornerofficeguy.wordpress.com) Data management professional (jparnitzke.wordpress.com) Essential Analytics (essentialanalytics.wordpress.com) The program office (theprogramoffice.wordpress.com) 45