Experimentation at Booking.com. Edoardo Valvo

Transcription

1 Experimentation at Booking.com Edoardo Valvo

2 Company highlights. 29 million listings including 1,550, M listing in room nights booked every 24 hours homes, apartments and other unique places to stay. 17,500+ affiliate partners 144 million + destination recommendations 1996 Founded in Amsterdam 30 different property types 24/7 149 million+ verified guest reviews online countries customer service supported languages offices in over in 220+ countries 17,000+ dedicated employees Became Booking Holdings Inc.)

3 Edoardo Valvo Talk to me about: - Project Management - GTM Applications - Ultramarathon Running Who Are We? Stamatina Sykara Talk to me about: - Marketing - Partner Engagement Campaigns & Analysis - Greek islands

4 Who Are We? Booking.com Affiliate Partner Programme Connect your business to the world s #1 accommodation website. 17,500 affiliates, 3,000+ bloggers. Customisable and easy-to-use products Wordpress plugins. Dedicated commercial, optimization and technical support.

5 What is Experimentation? What they say about experimentation at Booking.com: [Their] utilization of A/B testing to determine what drives higher conversion [is] leveraged across its entire platform, resulting in conversion levels 2-3x the industry average. - Evercore Equity Research, 2013

6 If it can be a test, test it. If we can t test it, we probably don t do it. Stuart Frisby, Director of Design

7 2011

8

9 Experimentation Methodology

10 Problem Identification e.g. Top 10 things to do in Rome blog post. For each thing to do there s a button redirecting to the website for the event. My readers land on the post but don t click on the buttons. Scenario and Hypothesis To understand the scenario, I need to know how the readers access the page and where they re supposed to go after reading the post. I then assume that because my users land on the blog post, they simply don t find the button engaging. I will then test a change to the design of the button, to see if it will attract more attention. I expect that having a different look and feel will increase the number of clicks and time spent on my page, meaning users will engage further with the page s features. Goals and Success Metrics The results that indicate if the test is positive or not will be based on: Number of Clicks: +20% Positive Engagement: + 1 min time on page

11 A/B Testing

12

13 A/B testing is comparing two versions of a product/feature to see which one converts better. Website Website OR Buy now? Buy now

14 Buy now Coin flip 50% 50% Buy now

15 Website 50% 10% conversion lose 20% conversion win Buy now Website 50% Buy now

16 A/B Test on Copy Scenario: Customers wait until the last minute to book a deal. Hypothesis: Increasing urgency and clarity will increase the click-through rate (CTR). Goals: Increase in CTR and ultimately more bookings.

17 A/B Test on Copy Description: Highlighting free cancellations on the Pay Local website helps to get more customers through the first step. Increase in bookings per day Positive effect on conversion A B

18 A/B Test on Design

19

20

21 A B

22 INCONCLUSIVE

23 Other testing methods

24 The complexity of A/B Testing A/B testing becomes trickier if you: want to test a very small effect change (%). select a variable with a very limited effect on the page. don t have a lot of time to run an experiment. want to test with maximum statistical precision. need to perform a highly complex test.

25

26 All hail simplicity Live Website Testing Website Only one website version is used. 100% of the traffic is directed to the same page. Experiment run-time needs to be long (4 weeks). You can test one main component of the page each time. Metrics to be tested have to be limited (max. one or two metrics). Select a component that has a clear effect on the page s look and feel (e.g. CTA copy, highlighted content, links, pictures, colors). Website 10% conversion Buy now Version A : 4 weeks Buy now 20% conversion Version B : 4 weeks

27 Assumption & Hypothesis 15% of my visitors are from Germany. I believe these users want to read my content in German, and by offering my blog in German, these users will interact with my site content for longer. How to validate Think about how you can test your assumption by defining a hypothesis. e.g. I believe if I add an option to change the language to German, the bounce rate for this market will decrease. Goals and Success Metrics Validate your assumptions 10% more German visitors click on a new German language option before leaving. Goals Goals and Success Metrics nd Success Metrics

28 Final tips on Experimentation

29 Methodology is everything Trust the process Identify, list, prioritize and track 1 Select tools that you can trust Test atomically and consistently Keep the test context neutral

30 Keep your content clear and simple Be clear People have limited time and attention spans Use the present tense and the active voice Be minimalistic, think mobile first Choose the simplest form of a verb that can be universally translated and understood. 2 Positive effect on: User Experience, Content Engagement, Impressions.

31 Keep your content clear and simple

32 3 Keep Your Content Concise Write short sentences, make them scannable Break the message into chunks If there is a shorter way to say what you mean, do it Be aware that space is limited Positive effect on: CTR, Engagement with non-text features

33 Keep your content consistent Avoid mixing "me"/"my" with "you"/"your Pay attention to capitalization and punctuation Check the context. Be a detective first, an artist later 4 Positive effect on: CTR, Engagement with non-text features

34 Keep your content consistent

35 Content: Pre-Conversion User Experience Phases PLANNING SEARCH DISCOVERING DREAMING CONVERSION PURCHASING DECISION-MAKING

36 5 Content: Inspire your visitors Inspirational and playful copy appeals, especially when people are planning their next trip Objective is to boost their ego Personalization also works well Positive effect on: Clicks, Time on page, Pageviews, Engagement

37 Content: Persuade your visitors Persuasion works well in the decision-making phase, where travelers can be overwhelmed by several options. Provide the right information, at the right time Use methods like social proof (positive comments), urgency and scarcity 6 Positive effect on: CTR, Content Engagements

38 7 Content: Reassure your visitors Reassurance works best in the final-decision phase Clarify what s included in an experience e.g. Identify a reader s potential concerns and address them in your post or the comments section Positive effect on: Shares on Social Media, Content engagement

39 Let s wrap it up! 1. Testing is a mindset 2. A/B testing is only scary if you don t know what you want to achieve 3. There are simple ways to test that anyone can use 4. A test is not forever

40 Talk to us at the Booking.com booth to: Get access to our digital TBEX Media Kit Sign up today and get access to 40% revenue share! Grazie mille.