Experimental marketing: why and how Andrey Sebrant, Director, Product marketing Sofia, October 2010
A few facts about Yandex Over 50 million users a month, around 15 million a day Over 100 million search queries a day Over 64% of search traffic in Russia Over 2 600 employees in 7 offices across three countries Over $300 million in revenues last year 2
A very competitive environment What do we rely on: Dream team Big math Computing super power Last but not least: a very efficient marketing 3
Digital means a lot Of course we operate in a digital media We collect so many data that the entire marketing becomes digital in its core It takes many skills to be a pilot, much more than just ability to read devices. On the other hand, if you don t know how to read them, the landing may be too hard. 4
Marketing is an experimental quantitative science Marketing studies happiness of users and customers. We have to predict what makes them happy even if they do not realize where the happiness is. We study the world (people, products and markets) by creating models and testing them in experiments Models are based on already known facts, observations, intuition and fantasy Experiments or tests are carried out using standard scientific methods 5
Product is the king On the Net this is true as nowhere else: - everything is free - all competitors are a few clicks away 6
Case study: Yandex bar Hypothesis: short search window results in fewer searches because search queries are getting longer Experiment: 50% of new bars have longer window. Tracking: search activity, search queries length 100% 186% Test bars/control bars 104% 105% 103% No. of searches Query length, words Query length, chars Churn Rate Yes, size matters, but not because of the query length 7
Case study: Churn rate 360000 310000 Loss of users may look scary Don t panic! Analyse this It turns out that users are similar to isotopes; they decay like in nuclear physics 260000 210000 160000 110000 60000 10000 1 2 3 4 5 6 7 Number of users registered during week 1 and visiting the service at least once during each next week 8
Case study: Churn rate 1000000 360000 310000 Loss of users may look scary Don t panic! Analyse this It turns out that users are similar to isotopes; they decay like in nuclear physics 260000 210000 100000 160000 110000 60000 10000 10000 1 1 2 2 3 3 44 5 6 77 Number of users registered during week 1 and visiting the service at least once during each next week 9
More about radioactive decay The same percent of users quit service every week 1000000 100000 y = 59125e -0,06x 10000 Decay rate is characterized by half life, the time taken to decrease the initial amount of users by half 1000 100 10 If the process is described by the sum of two exponents, there are two groups of users with different half lives 1 y = 1E+06e -1,341x 1 2 3 4 5 6 7 8 9 10 In many practical cases the churn rate curve is the sum of two exponential decay curves 10
Translation from mathematics to marketing Two groups mean not two isotopes in our case: we observe loyal (long half live) and casual users Thus, we get quantitative metrics for loyalty which we can measure at the early stages of a new (or relaunched) service. Why is this so important? 11
Real-time obsession makes us blind to long-term effects The internet is too fast We all are obsessed with instantaneous measurements - «At the focus group they say» - «usability tests show» - CTR is dropping! And we often forget about life-long relationships with and love of our users ;) 12
Source data: tables of registrations and visits Week Week 01, % Week 02, % Week 03, % Week 04, % 19.07.2010-25.07.2010 100,00 45,93 35,86 32,58 12.07.2010-18.07.2010 100,00 51,23 40,36 34,91 05.07.2010-11.07.2010 100,00 61,93 49,25 44,54 28.06.2010-04.07.2010 100,00 59,55 50,17 43,97 21.06.2010-27.06.2010 100,00 61,13 48,75 43,46 Every week you need a report on how many users registered in previous weeks are still using a service. Also, it s a useful alerting tool. 13
Churn rate is a good metric for evaluation interface changes You can measure the effect of interface updates on users loyalty. Ideally, the new interface should be tested on a small percent of users. Control group: new users registered in the old interface Test group: new users registered in the new interface Metrics to monitor: ratio of percent of loyal users, ratio of half lives And then you can predict the future! 14
Churn rate and the budgets (usability vs. advertising) Too much of the ad budgets are a waste 30000 25000 20000 15000 Ad campaign Loyal attracted = 100%, Half life is eternity Loyal attracted = 5%, Half life = 3 Most advertising campaigns are too far from the ideal, because they mostly attract casual users and very few loyal ones 10000 5000 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 15
No math, just common sense Before pouring expensive users into the website, plug the holes! 16
But if you like math Site audience in a month n depends on the audience of the previous month: A ( n) A( n 1) (1 C) N Decrease in churn rate costs money. But usually it s one-time expense: C G c C maх (0;1) G c ($); New users come on their own and thanks to promotion activities: N Nnat Fnew($) 17
Enter the segmentation Different groups of users Different ads Different sources of new users Test and measure responses in each group, test and measure efficiency of every source and ad And think it all over! 18
Not only digital things matter 19
Not only digital things matter This still is the most important tool for a marketeer 20
Thank you! Спасибо за внимание Andrey Sebrant e-mail: asebrant@yandex-team.ru twitter: @asebrant FB: www.facebook.com/asebrant