Collecting Data, Analysing Data and adjusting the product. Dr. Riku Suomela Mixed Reality Director Next Games

Transcription

1 Collecting Data, Analysing Data and adjusting the product Dr. Riku Suomela Mixed Reality Director Next Games

2 Today What I will cover: Software as a service Consumer products Collecting data to refine the product your options. What to measure at different stages of production, and what are your possibilities for corrective actions. Goal: to make succesful ROI positive SaaS consumer products.

3 Background of myself Location based and Mixed Reality application and games research Nokia N-Gage Lumia devices Mixed Reality Games Developed SaaS products for 15+ years.

4 Outline for today Concepting Prototyping Preproduction Production Market test Live service End of life Quantitative data Qualitative data

5 1. prototyping Concepting Prototyping Preproduction Production Market test Live service End of life Quantitative data Qualitative data

6 Main goal: validate an unknown It may be technical, user facing or any other problem The most important thing is to fail and succeed fast. Define what you are measuring, and measure only that (as everything else should be really bad )

7 How to collect data Qualitative data from tests with the team + potential customers

8 Main problem: Cognitive biases In the beginning of a production, there is very little useful quantitative data, but plenty of qualitative data. An example: Confirmation bias makes us interpret the results as if they validate our hypothesis and assumptions.

9

10 2. Preproduction Concepting Prototyping Preproduction Production Market test Live service End of life Quantitative data Qualitative data

11 Main goal: define the final product in order to execute fast This phase still contains a lot of prototyping, but now the complete product is prototyped. The product should be iterated fast and now tests should be complementing each other.

12 How to collect data Qualitative data from tests with the team Several different features are being tested àtest A1 (hypothesis 1) à Test A2 (hypothesis 2) àtest B1 (HB1) à Test B2 (HB2)

13 Main problem: lack of qualitative data, lack of a complete product At this point, most of the final product features are done separately Each part can be tested separately, but they most often don t work great together. à it is difficult to know is the final product great compared to it having great features.

14 3. Production Concepting Prototyping Preproduction Production Market test Live service End of life Quantitative data Qualitative data

15 Main goal: define the minimal product to test in the market Execute the minimal set of features to get quantitative data. Aim for: Minimum Viable Product. However, the requirements for a minimum viable product keep changing daily, especially for a highly competed market. Example: Farmville (Facebook game, 2009) was developed in 5 weeks. Today, a minimum viable facebook game would take a multiple of that (as market has evolved).

16 How to collect data Main goal in production is to instrument everything in the product for measurement. If it is relevant, you can collect every press of a button Several products exist in the market, that can be used. As before, qualitative data from user tests can and should be constantly collected from the team, but now everything must be instrumented so real usage data is collected.

17 Main problem: lack of real users The product is getting mature, but still qualitative data is key. Still, it pays to be fast, and get to the market.

18 4. Market test Concepting Prototyping Preproduction Production Market test Live service End of life Quantitative data Qualitative data

19 Market test types Testing the market may be purely technical (does the system work as intended). This is an extended QA period. Test maybe open or closed depending on what you test. Closed beta (invite only users) allows more freedom, as it is beta. Open testing (open in a distribution channel) gives you less freedom but it gives you real data on real customers.

20 Defining metrics (KPI = Key Performance Indicator) Retention (1,7,30,60,90,260) how long the people stay in your product? The First Time User Experience (FTUE) Funnel how far the people get before churning. Churn people abandoning your product. LTV (Life Time Value) how much money per user is being made. LTV = ARPDAU * average time a user spends on your product. This changes daily! Daily Active Users (DAU). Similarly MAU (monthly). ARPDAU = Average Revenue Per Daily Active User. CPI = Cost Per Install (how much does a single new user cost) ecpi effective CPI (including organic users, i.e. User who were not acguired with money) Conversion = how many people convert to paying customers (only applies to free apps and games)

21 Main goal: this is it. Is the product going to be profitable? Test your minimum viable product. If (LTV > ecpi) à you are profitable. But you don t know this at the beginning. What to optimise, and in which order 1. FTUE. Make sure users understand the basics. 2. Retention. Make sure your users stay. One day at a time. 3. User Acquisition. Make sure you don t pay too much for new users (CPI) 4. LTV. Make sure you make more money per user than CPI.

22 How to collect data Everything is instrumented and data is pouring in. How is no longer the relevant question.

23 What to do with the data. This is the question.

24 Main problems: now there are plenty! The cognitive biases are still there. Data can validate your hypothesis if you are not careful. Golden cohort - the people who install your app first. They perform significantly better than subsequent users, as they were interested in the first place. People may ignore 60% of your app you get no data! Some technical problems contribute to the problems, but may leave no trace, as that may be the last thing a user does.

25 How much data is needed? Use: Depending on your product, aim for the first 10, 100, 1000, 10000, users. Depending on your product, aim for a sustained DAU of 10, 100, 1000, (This can be increased gradually) Retention: It takes 30 days to get the first set of D30 retention data. Be patient. Often, your problems are in the first minutes

26 Gaming example

27 Landing page (marketing) App Store Install Start D1 D7 D30 Your target Median conversion date LTV (D60)

28 Landing page (marketing) App Store Install Start D1 May look like this Median conversion date

29 Which means you must start from the very beginning How can I get the people to START the game? How can I get people to FINISH THE TUTORIAL? How can I get people to get back on the first day to the game?

30 Reaction times Problem area Time for user to experience the problem Time needed to collect data Start of the app 0 seconds 1 Day Is there a technical problem? Tutorial 1 minute 1 Day Back to the drawing board and quick. D1 retention 0 days A week The game is not desirable at all? D3 retention 0-2 days A week The game lacks content D7 retention 0-6 days Several weeks The game lacks content D30 retention 0-29 days Over a month The game lacks content Note

31 Can this information be gathered prior to market test? Not reliably. All testing prior to a real market test has biases. Players who are recruited by external testing companies tend to like the testers and hence the game. Expert evaluations are exactly that: they are not consumers. Internal company testing is not your customers.

32 All is not lost Be diligent. Address ALL ISSUES, one issue at a time, start from the beginning. It is a funnel à if there is a problem earlier, don t focus on later ones. They may be working already!

33 Landing page (marketing) App Store Install Start D1 D7 D30 Start from this Median conversion date LTV (D60)

34 Landing page (marketing) App Store Install Start D1 D7 One issue at a time D30 Median conversion date LTV (D60)

35 Landing page (marketing) App Store Install Start D1 D7 D30 Make this happen Median conversion date LTV (D60)

36 5. Live Service Concepting Prototyping Preproduction Production Market test Live service End of life Quantitative data Qualitative data

37 If you are here, the following happened: 1. FTUE is working. 2. Retention is working. à players stay in your product. 3. User Acquisition is cost efficient. à you can market your product. 4. Life Time Value is healthy (and maybe growing?) à You can market and get new users.

38 You are here the longest time. Lifetime of a service is counted in years, this is where development starts for real. Development prior to being in market is as short period (it does involve your entire knowledge though). Aim to get here as fast as possible. It makes more sense to abandon a product and start building on it, than work on a product for years if KPIs are too low.

39 Landing page (marketing) App Store Install Start D1 D7 You got here! Now make it better D30 Median conversion date LTV (D60)

40 Something to think about what shape is the funnel Product Average lifetime Average Revenue Per User Netflix Spotify Clash of Clans Pokemon Go <add any> Average Revenue Per Paying User

41 If you got here, the most important rule: Don t make it WORSE You have a profitable product, it is very easy to lower all numbers! All new features may lower the OVERALL performance of the product, eventhough increasing certain parts of it. Again, test it! But now you need a new tool.

42 A/B testing Your baseline is the current product Your new feature is A Another version is B Compare the performance

43 A/B testing cohort size You need a statistically relevant size for your new feature, how many people do you need? A LOT! And you need to be careful to put in the right population. This is extremely hard to get right, and is the topic of a separate lecture!

44 A/B testing process Live Product Performs? Development New Feature in market A B Compare Analyse Data

45 New feature should not always be released A new feature maybe released if it Improves at least one KPI Does not lower other KPIs.

46 What if a feature stops performing? It can be A/B tested out.

47 Effects on development Your backlog can be miles long, filled with super features! None of it matters, if users cannot get far enough in your service. All features need to be tested in market, before releasing to not lower overall performance.

48 Conclusion Live service Concepting Prototyping Preproduction Production Market test End of life Quantitative data Qualitative data

49 Conclusions 1. Drive to market fast and test the product in market (Soft Launch). You can be a long time in this phase. 2. Trust your data. When you have enough, it tells the right answers. 3. Prioritise for a short period of time. Data tells you where to focus. 4. Optimise all KPIs (whatever they are). 5. Once you have a succesfull product, aim to keep it there (with A/B tests). 6. Your product spends more time in market, than in development.

50 Thank You