From Theory to Data Product

Transcription

1 From Theory to Data Product Applying Data Science Methods to Effect Business Change KDD August 13

2 Advanced Analytics Entry Points

3 Strategy Organization Policies, Procedures & Standards Components of an Advanced Analytics Project Corporate Drivers Insight Integration Data Strategy Change Management Data Driven Decision Making Architecture Business Value Planning Exploratory Analysis 3

4 IT What is an entry point? Bottom-Up Business Top-Down 4

5 5

6 IT Bottom-Up Business The Technology Directive Top-Down 6

7

8 Results were questioned and not aligned with current processes 8

9 Our vendor is really confident with this technology! 9

10 We do need Data Governance and we ll do that once we ve shown value 10

11 I m not sure who will look at it but it will be interesting 11

12 It s not that easy 12

13 The data is pretty self-evident 13

14 We didn t schedule time with [the SME s] for that 14

15 15

16

17 IT Bottom-Up Business The Technology Directive Top-Down 17

18 IT Bottom-Up The Field of Dreams Business The Technology Directive Top-Down 18

19

20 The right business people were not identified project resources 20

21 No executive support to move forward 21

22 We have easy access to the data 22

23 If we build it, they will come 23

24 I can t tell you what that means, you need to talk to 24

25 I can t get time with [SME] to 25

26 What happens if this information gets out? 26

27 Why wasn t I involved? 27

28 28

29

30 IT Bottom-Up The Field of Dreams Business The Technology Directive Top-Down 30

31 IT Bottom-Up The Field of Dreams Business The Technology Directive The Ambitious Executive Top-Down 31

32

33 End users saw no value in, or use of, the results 33

34 We don t need to talk to the end users 34

35 It doesn t matter what our current metrics are 35

36 We only use standard industry terms 36

37 We are just implementing what the business tells us 37

38 Are we sure this is what the users want? 38

39 39

40

41 IT Bottom-Up The Field of Dreams Business The Technology Directive The Ambitious Executive Top-Down 41

42 IT Bottom-Up The Field of Dreams The Smart Competitor Business The Technology Directive The Ambitious Executive Top-Down 42

43

44 We don t know what we don t know 44

45 The data is self-evident 45

46 That s third party software. We don t have a data dictionary 46

47 Doesn t seem like they are very interested 47

48 48

49

50 IT Bottom-Up The Field of Dreams The Smart Competitor Business The Technology Directive The Ambitious Executive Top-Down 50

51 IT Bottom-Up The Field of Dreams The Smart Competitor Business The Technology Directive The Ambitious Executive Top-Down 51

52 IT Bottom-Up The Field of Dreams The Smart Competitor Business The Technology Directive The Ambitious Executive Top-Down 52

53 53

54 IT Project Start Identify Business SMEs Identify Business Drivers Discuss business SME resource gaps with PM/Project Sponsor The Technology Directive Top-Down Midstream Project Start Gather results to date Engage business SMEs in workshop to focus on actionable insights 54

55 IT The Field of Dreams Bottom-Up Project Start Identify Business SMEs Identify Business Drivers Talk to business counterpart to garner support and participation Midstream Identify proper data stewards Engage business SMEs in workshop to focus on actionable insights 55

56 Project Start Identify Actionable Business Question Midstream Talk to end business users to best understand process and fit Business Identify Business Drivers Top-Down The Ambitious Executive 56

57 Bottom-Up Project Start Engage C-Suite early The Smart Competitor Check in with business drivers and strategy Business Midstream Realign if necessary 57

58 IT Advanced Analytics Entry Points Bottom-Up Business Top-Down 58

59 Group Exercise 59

60 Are you asking the right questions?

61 Here s what we re trying to avoid 61

62 Here s what we ll talk about What is a valuable business question? How do you define criteria specific to your business to identify valuable business questions How do you frame those questions and support prioritization Before and After examples of questions that have gone through this process

63 What is a Valuable Business Question?

64 When answered, a specific action can be taken which provides a measurable result in line with strategic business objectives.

65 Our Method - Activities Discovery Release Planning & Foundation Sprint 1 Sprint N-1 Sprint N Project Kick-Off Create Project Backlog Experiment Design Present Final Results Driver Definition Workshop Define Project Schedule Experiment Implementation Next Steps Coaching Business Value Workshop Define Architecture Experiment Testing Data Asset Inventory Install & Config Platform Analysis Prioritization Map Initial Data Ingest 65

66 Driver Definition Workshop

67 Business Value Workshop Who to invite? What do they need to know before they arrive (prepping them in advance)? What do they bring? 67

68 The First Hour

69 Non-Starters Can we show value with tool X? Data science helps businesses how can it help ours? How can we establish KPIs to more clearly indicate when high-level actions should be taken?

70 Preconceived We need to use deep learning to do some stuff Produce an algorithm that can be used, season to season, to set prices when new products become available in the market from us or competitors Create a full list of subscribers and their respective churn probabilities, indicating probability to churn within 4 weeks 70

71 Requires Refinement Who are the biggest users of healthcare in? How can we save $ on our preventive maintenance? When will customers leave? What products / features should we build?

72 Hour 2 and Beyond

73 Refining (Workshop) Brainstorm (Workshop!) Evaluate (Map!) Refine (Key Questions!) (Map) (Key Questions!) 73

74 Scenario 1: Telecommunications

75 In the business value workshop STARTED HERE: We want a machine learning model that predicts customer churn four weeks in advance of the event, with an accuracy of at least 80% and lift curves graphed to demonstrate why you picked this model. 75

76 We want to know when our customers will leave. Well, it doesn t always. Only some of the customers leaving actually matters.

77 High-value customers matter when they leave, they take a lot of business with them. Um Predicted lifetime value = $x/time unit/y years Well, we ll make sure customer service knows so they re careful with people who call in with a complaint. And maybe we ll offer them something to make staying more appealing. A special offer we d need at least two months to put something like that out. And how long do we need to change customer service approaches before someone leaves? Can we predict which of our high-value customers will leave in two months?

78 Candidate Business Question Can we predict which of our high-value customers will leave in two months? 78

79 Scenario 2: Healthcare

80 In the business value workshop STARTED WITH: Which of our company s three options is the most effective treatment approach for a given patient? 80

81 Why? Who? What? Where? Do we have it?

82 Candidate Business Question How do you predict the optimal treatment for any given patient before treatment is prescribed? 82

83 Evaluating and Prioritizing

84 Our Method - Activities Discovery Release Planning & Foundation Sprint 1 Sprint N-1 Sprint N Project Kick-Off Create Project Backlog Experiment Design Present Final Results Driver Definition Workshop Define Project Schedule Experiment Implementation Next Steps Coaching Business Value Workshop Define Architecture Experiment Testing Data Asset Inventory Install & Config Platform Analysis Prioritization Map Initial Data Ingest 84

85 Let s dive in to one example 1. How do you predict the optimal treatment for any given patient before treatment is prescribed? 2. What other products should we develop as part of our existing product line? (predict market needs) 3. Can we compile data from our healthcare providers to understand optimal practices in installing our products based on outcomes (recovery time)? 4. Can we determine poor performing products before they leave the factory? 85

86 Prioritization Criteria Value Feasibility Alignment 86

87 Value Value Timeframe Value Timeframe 10 8 Alignment Score Value Score Value Score 0 Alignment Timeframe Feasibility Timeframe Feasibility Score 87

88 Feasibility Value Timeframe 10 8 Alignment Score Value Score 0 Alignment Timeframe Feasibility Timeframe Feasibility Timeframe Feasibility Score Feasibility Score 88

89 Alignment Value Timeframe 10 Alignment Score Alignment Score Value Score 0 Alignment Timeframe Alignment Timeframe Feasibility Timeframe Feasibility Score 89

90 Prioritization for 0ur scenario Value Timeframe 10 8 Question 1 Alignment Score 6 4 Value Score Question Question 3 Question 4 Alignment Timeframe Feasibility Timeframe Feasibility Score 90

91 Valuable Business Question: When answered, a specific action can be taken which provides a measurable result in line with strategic business objectives. Brainstorm (Workshop!) Evaluate (Map!) Refine (Key Questions!) 91

92 Group Exercise 92

93 Agile Approach to Data Driven Decision Making

94 Strategy Organization Policies, Procedures & Standards Components of an Advanced Analytics Project Corporate Drivers Insight Integration Data Strategy Change Management Data Driven Decision Making Architecture Business Value Planning Exploratory Analysis 94

95 What is Agile? An iterative and incremental approach to project management and delivery Originated for software development Adapted for data driven decision making projects Allows project teams to adapt quickly to changing requirements and/or new insights 95

96 Agile Principles Source: Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. Business people and implementation team must work together daily throughout the project. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. Working software is the primary measure of progress. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. Continuous attention to technical excellence and good design enhances agility. Simplicity--the art of maximizing the amount of work not done--is essential. The best architectures, requirements, and designs emerge from self-organizing teams. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. 96

97 Managing Uncertainty Experiment design is focused on constant progress toward the stated goal By their nature, experiments may not yield expected results Project strategy should be to adapt subsequent experiments, based on findings, to continue progressing toward the goal Using this approach mitigates risk Key activities early in the project validate feasibility of moving forward Goal is to fail fast reassess throughout to avoid unproductive activity 97

98 Agile Approach Charter Captures business drivers, guiding principles, scope, approach, who is involved Themes Prioritized Business Questions for investigation Epics Collections of Stories that comprise an investigation Theme Stories Units of work that can reasonably be completed in a Sprint 98

99 Other Agile / Scrum Terms Product Backlog An ordered list of everything that might be needed in the project Single source of requirements for any experiments to be conducted Sprint Backlog The set of Product Backlog items selected for the Sprint, plus a plan for delivery A forecast about what experiments will be addressed in this Sprint and the work needed to deliver these experiments 99

100 Our Method - Activities Discovery Release Planning & Foundation Sprint 1 Sprint N-1 Sprint N Project Kick-Off Create Project Backlog Experiment Design Present Final Results Driver Definition Workshop Define Project Schedule Experiment Implementation Next Steps Coaching Business Value Workshop Define Architecture Experiment Testing Data Asset Inventory Install & Config Platform Analysis Prioritization Map Initial Data Ingest 100

101 Our Method - Activities Discovery Release Planning & Foundation Sprint 1 Sprint N-1 Sprint N Project Kick-Off Create Project Backlog Experiment Design Present Final Results Driver Definition Workshop Define Project Schedule Experiment Implementation Next Steps Coaching Business Value Workshop Define Architecture Experiment Testing Data Asset Inventory Install & Config Platform Analysis Prioritization Map Initial Data Ingest 101

102 Types of Experiments Focused on data within each sprint Ex. Feature analysis Focused on testing results of a model Ex. Cross validation, lift, gain, etc. Focused on testing the insight A/B Testing, Focus groups, etc. Focused on data product Checking for model drift, model maintenance Ex. Recommendations Engine, etc. 102

103 Each Sprint Includes: Sprint Planning Workshop Updates to Product and Sprint Backlogs Development Activities Additional data ingestion <optional> Analysis & Review of Progress Check-in with Strategy & Drivers 103

104 Our Method - Deliverables Discovery Release Planning & Foundation Sprint 1 Sprint N-1 Sprint N Business Questions Document Prioritization Map Architecture Document Development Environment Setup Test Dataset Analysis Findings Final Results Recommended Next Steps Data Dictionary Project Schedule Data Quality Assessment Backlog Project Charter 104

105 Our Method Team Members Please Note: An individual may play more than one role Discovery Release Planning & Foundation Sprint 1 Sprint N-1 Sprint N Project Manager Project Manager Project Manager Project Manager Data Science Strategist Data Science Strategist Data Science Strategist Data Quality Analyst Data Quality Analyst Workshop Facilitator Data Architect / Developer Data Architect / Developer Product Owner Product Owner Product Owner Data Science Team 105

106 Responsible Accountable Consulted Informed (RACI) Deliverable Architect Data Science Strategist Data Quality Analyst Data Science Team Designer / Developer Business Questions Doc I RA C I I I Product Owner Project Manager Prioritization Map RA I I I Data Dictionary A R I I I Data Quality Assessment A R Project Charter C C C C C C RA Architecture Document RA C C Development Environment Setup A I R I I Project Schedule C C C C C C RA Backlog C C C C RA I Test Dataset C C RA I I I Analysis Findings A R I I Final Results C RA C I I Recommended Next Steps R RA C C I 106

107 Group Exercise 107

108 Strategy Organization Data Strategy Policies, Procedures & Standards Thank You! Corporate Drivers Change Management Business Value Planning Insight Integration Data Driven Decision Making Architecture Exploratory Analysis Janet Forbes Lindsay Brin Danielle Leighton