How to make Agile "work" in Business Intelligence projects. Tom Breur VP Data Analytics, Cengage Learning San Diego, 19 April 2016, 11:15-12:00

Transcription

1 How to make Agile "work" in Business Intelligence projects Tom Breur VP Data Analytics, Cengage Learning San Diego, 19 April 2016, 11:15-12:00 1

2 Presentation overview How to make Agile work My experience with (Agile) BI projects What matters (most) for success? in Business Intelligence projects Applying Agile in BI is fairly new What is (truly) unique about applying Agile to BI? With an eye to: People/process BI architecture 2

3 Part 1: People & Process 3 3

4 Our highest priority is to satisfy the customer through early and continuous delivery of valuable software Agile Manifesto, 2001 Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, Dave Thomas 4

5 Our highest priority is to satisfy the customer through early and continuous delivery of valuable software Agile Manifesto, 2001 Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, Dave Thomas 5

6 Quick & Dirty Agile Technical debt stands squarely in the way of continuous delivery, and maintaining a sustainable pace technical debt creates (new) legacy! 6

7 Agile is "just" common sense, or is it? Agile, self organization What managers (sometimes) think People are more productive when nobody tells them what to do An efficient team must have slack, must have people sitting idle, with nothing productive to do, on a regular basis Only the business has the right to choose what gets done Pair programming leads to more (effective) production code OMG! Shouldn t anybody be giving them instructions?? Isn t there work to do?? Don t just stand there do something!! You mean its OK if they just oodle?!? You re kidding, right? Our product owners don t know this software very well, they need our help What s going on?!? Why isn t that other guy doing anything?!? Are we actually paying for him?? 7

8 Simplified Development Error Reports Idea Develop Test Working Code 8

9 More realistic development cycle(s) Idea Analysis Design Code Error Error Error Working Code Acceptance Test System Test Unit Test 9

10 Minimize software inventory Work in Progress is a liability not an asset 10

11 Cost versus Throughput Accounting Cost Accounting Throughput Accounting (ToC) Inventory is an asset Efficiency = function/ dollar (hours) labor is a variable cost People sitting idle are discarded! Inventory is a liability Efficiency = function/ direct costs (idle or not) labor is a fixed cost People sitting idle are part of the system! 11

12 Part 2: BI Architecture 12

13 Why go Agile? (1) BI projects fail too often, or don t live up to expectations Increasingly, BI development takes place alongside (instead of after) application engineering 13

14 BI requirements are inherently unstable Information products trigger change requests: new data insights new requirements Gerald M. (Jerry) Weinberg: Without stable requirements, development can t stabilize, either 14

15 Weinberg on Quality If quality isn t an objective (if the software doesn t have to work), you can satisfy any other constraint (e.g.: budget, time, etc.) Gerald M. (Jerry) Weinberg 15

16 BI: means and ends uncertainty Means uncertainty How do we get there? Lack of design patterns Data integration fraught with data quality issues Lack of Master Data Management Lack of Meta Data No agreement on how to conform dimensions Ends uncertainty Where are we going to? Requirements are difficult to pin down Diverse end user groups Ambiguous business case(s) Scope is unclear Data warehouses are never done 16

17 Waterfall Agile Waterfall/Traditional Agile Requirements Resources Date Fixed Plan Driven Value Driven Estimated Resources Date Requirements Agile fixes date & resources and varies the scope source: Dean Leffingwell (2011) 17

18 Concurrent development (1) Waterfall: you can avoid mistakes/rework by getting good requirements upfront The most costly mistakes arise from forgetting important elements early on Detailed planning (BDUF) requires: early (ill informed) decisions uses more time leading to less tangible products to resolve ambiguity 18

19 Concurrent development (2) Agile: decide at last responsible moment decisions that haven t been made, don t ever need to be reverted No free lunch deferring decisions requires: anticipating likely change coordination/collaboration within team close contact with customers 19

20 Inmon Kimball (1) Hub-and-Spoke 3-tiered Bus matrix 2-tiered 20

21 Inmon Kimball (2) Problems with Inmon Uncovering the correct 3NF model requires scarce business expertise Unclear where 3NF model boundaries begin and end Model redesigns trigger a cascading nightmare of parent child key updates Problems with Kimball Smallest unit of delivery is a Star, and incremental growth adds prohibitive overhead Dimensional structure is very rigid not conducive to expansion or change Conforming dimensions is hard, especially without access to data 21

22 Horses for courses 3NF quickly & accurately capture transaction data easy to get data in Hyper normalized (Data Vault) integrate historical data capture all data, all the time Dimensional present & analyze data easy to get data out 22

23 Agile & Data Vault (Hyper Normalized Models) Source to target mapping implies inherent sequential processing to maximize value creation from data (very) few other architectures allow incremental build at such low marginal cost Deliver early in (very) small increments (very) few other architectures allow mistakes in your model, that you can recover from inexpensively Deliver early (long) before you have settled on the final business model 23

24 Data modeling adds value value dimensional Data Vault ODS staging time 24

25 3-tiered DWH architecture Metadata Legacy Datamart 1 OLTP Datamart 2 ERP LOG files ETL Staging Area Data Warehouse Datamart n Business Intelligence Applications ODS External 3 NF hyper normalized dimensional 25

26 Backroom Frontroom Back room Data Warehouse Architecture Legacy Metadata Front room Business Intelligence Architecture Datamart 1 OLTP Datamart 2 ERP LOG files ETL Staging Area Data Warehouse Datamart n Business Intelligence Applications ODS External 26

27 Divide & Conquer Break down semantic gap between back- to front room Offer a range of data services: Source data as is (aka ~ Persistent Staging) Some modeling and partly historicized (aka ~ ODS) Lightly modeled and fully historicized (aka Data Vault, Anchor Modeling, etc.) Source data that have undergone cleansing Dimensional models Full-fledge BI applications Allow business to set priorities! Where should time and effort be spent, throughout the various project stages? 27

28 Why data virtualization? Operational BI calls for real-time data Integrate heterogeneous sources, at least in the eye of the beholder Data virtualization layer hides complexity about underlying applications & enables sharing of meta data Data virtualization enables federation, so you can delay (definitive) modeling, yet make data available early 28

29 Conclusion Go as fast as you can but no faster Beware of the placating dynamic Teams can maximize throughput ( Velocity ) by swarming around tasks on the critical path Swarming serves to minimize WIP! Requirements change get over it The sequential nature of BI value creation makes it extra important to keep an eye on minimizing WIP Dimensional models do not lend themselves well for incremental delivery: Either all data remain available in a persistent staging, Or you need to lower the on ramp time for delivery (for instance by using hyper normalized data models 29

30 Appendix: example of reengineering impact on data models Badarinath Boyina & Tom Breur, 2013 Adapting Data Warehouse Architecture to Benefit from Agile Methodologies Soure: 30

31 3NF Dimensional (1) 31 31

32 3NF Dimensional (2) this problem gets (much!) worse with multiple parent-child levels see: Kimball design tip # esign-tip-149-facing-the-re-keying-crisis/ 32 32

33 Hyper normalized model (Data Vault) business keys, context attributes (history), and relations, all have their own tables appending Supplier data to the model (or any other new source), is guaranteed to be contained as a local problem (=extension) in the data model because business keys, context attributes (history), and relations all have their own tables 33