THE ULTIMATE GUIDE TO MOBILE APP A/B TESTING

Size: px
Start display at page:

Download "THE ULTIMATE GUIDE TO MOBILE APP A/B TESTING"

Transcription

1 THE ULTIMATE GUIDE TO MOBILE APP A/B TESTING

2 Contents Introduction: The Rise of App Demand and The Need for Analytics...01 How A/B Testing Can Supercharge Your App Analytics...04 What Can You A/B Test?...07 Tips and Tricks for Top-Notch App A/B Testing...12 Real-Life Case Study: RunKeeper...15 Winning in the Mobile Market...18

3 1 INTRODUCTION THE RISE OF APP DEMAND AND THE NEED FOR ANALYTICS

4 1 INTRODUCTION THE RISE OF APP DEMAND AND THE NEED FOR ANALYTICS Mobile app demand shows no signs of slowing down. In fact, the App Store and Google Play alone now boast nearly 1 million apps apiece, and over 56 million apps were downloaded last year. The old pay-per-download model has shifted to in-app efforts such as advertising, purchases, freemium and lead generation, while non-monetization goals like customer loyalty and driving market share have risen. These days, apps need to differentiate from the crowd, and monetize while doing it. According to a recent Duke University study, the majority of marketers feel the pressure to prove ROI in every channel, including mobile. So to stand out in a competitive space in which millions of options beckon, marketers must know with certainty what is working and not working within their brand s mobile app, and use that knowledge to create more successful, engaging interactions Introduction: The Rise of App Demand and The Need for Analytics

5 Testing and identifying the right analytics is key to eliminating app guesswork and realizing success. Marketing is no longer a go-by-your-gut business, and we think has been replaced with we know. But testing mobile apps has historically been more complicated than testing a website or an , both in scale and in execution. And when it comes to development alone, you don t want to waste precious time needlessly altering code and re-uploading to the App Store. In this ebook, you will learn how automated app A/B testing gives you the freedom to trial ideas and hypotheses within your app, on-the-fly and without making drastic development changes Introduction: The Rise of App Demand and The Need for Analytics

6 2 HOW A/B TESTING CAN SUPERCHARGE YOUR APP

7 2 HOW A/B TESTING CAN SUPERCHARGE YOUR APP ANALYTICS Mobile apps may be booming, but they also are, and always have been, an entirely different platform than websites, particularly when it comes to measurement. Traditional online marketing metrics, such as pageviews and clicks, simply don t apply here. Marketers need to track app-specific analytics that focus on user behavior, engagement and conversion rate in order to create an experience that users love and want to return to. Simple but scientific A/B tests can take your app analytics to the next level by proving what works or doesn t work within your app. Simple but scientific A/B tests can take your app analytics to the next level by proving what works or doesn t work within your app. Long used by marketers across other channels, an A/B test presents two different versions of a message, feature or piece of content to the same population segment. Because of complex code changes and re-submission requirements, A/B testing of mobile apps once was a slow, arduous process requiring data analysts and developers. But thanks to automated platforms, mobile app marketers now have the opportunity to run A/B tests quickly and easily and get immediate, actionable insights How A/B Testing Can Supercharge Your App Analytics

8 A/B tests provide the perfect opportunity to bring mobile analytics full-circle. For example, a Fortune 500 ecommerce brand created and evaluated two segments within their app: those who performed a brand name search versus those who did generic searches, and then compared their respective lifetime value (LTV). The results were clear: users who made brand-specific searches resulted in 154% greater average LTV. The company was then able to take immediate action to improve the app s design and search functionality to make brands as prominent as possible. With further A/B testing, the company was able to try new features and UI changes in controlled tests. Without the need for code alterations or app store re-submissions, they were able to perform these tests, measure the results and iterate product design before launching to all users. Using automated testing and actionable analytics resulted in their mobile apps contributing to increasing total company revenue by nearly 20% and net income by 25%. When your end goal is to increase the usage and profitability of your app, ready-to-run A/B testing is crucial to driving impactful marketing campaigns for your user segments and optimizing lifetime value How A/B Testing Can Supercharge Your App Analytics

9 3 WHAT CAN YOU A/B TEST?

10 3 WHAT CAN YOU A/B TEST? App marketers always encounter the question What do I measure? As we ve covered, traditional web analytics metrics don t apply in the app world. While every business model is different and companies may focus on different goals, there are certain app metrics that most marketers will want to address, including usage patterns, time spent in different parts of the app, session length and conversion rates. Three different types of A/B testing opportunities within mobile apps will help you hone in on the metrics that count. In-App Messaging In-app messages are contextually relevant messages sent to users while they are in your app, and are a great way to communicate with users as part of their app experience to improve engagement and retention. For instance, you can target super fans (your most engaged set of fans) with a message encouraging them to rate your app, or you could send out a creative thank you message. A/B tests can help determine what makes a successful in-app messaging campaign using data and user feedback, instead of making assumptions. Take this example: as a news organization, you want to prompt free users to upgrade to a paid subscription. You can start by creating a custom segment of users in which everyone has: Read the US edition of the publication Started a session at least 6 times in the last 3 days Read at least 8 articles in the Politics or World News sections in the last 3 days What Can You A/B Test?

11 How to Use In-App Messaging and Push Messaging Together Once the A/B test is launched, you can monitor closely which version is most effective at pushing fence-sitters to upgrade and then refine your in-app messages according to the results, or conduct further testing around the content, context and timing of the winning version. Push Messaging Push messages are notifications sent to users on their home screen when they are not in the app. This is a great option for re-engagement efforts -- that is, to drive users back to your app if they are not actively engaged with it or have fallen out of a desired funnel. But content, frequency and timing have to be done right when it comes to push messaging, or it can alienate and frustrate users. A/B testing can help you determine which push messages perform best -- with the right copy, engagement points and message structure What Can You A/B Test?

12 Here s a push messaging example for a retail app: Let s say a department store launches a push messaging campaign within its ios app to promote a highly sought-after new line of handbags. For this promotion, the retailer sends an offer to those who have previously shown interest in handbags for 20% off their first in-app purchase. The goal is to pique their interest and drive in-app handbag purchases. The retailer could run an A/B test with two versions of the push messaging campaign to significant samples of its target audience to see which message yields more conversions. The results? 20% of those customers who received Message A took the first step and opened the app while 15% of those who received Message B took the same action. At first glance, it seems that Message A wins the contest, because of its higher open rate. But the contest is not over yet; the end goal is for users to make an in-app purchase. If you were only tracking app opens from this campaign, you would continue to invest in Message A which may actually have a lower conversion rate to in-app purchase than Message B. Because of this unknown, it s critical to look at the full picture and purpose of the A/B test What Can You A/B Test?

13 App features and content Different app types have various feature and content elements that lend themselves to different triggered events marketers can track. An event could be viewing an article on a news app, or adding an item to a cart on an mcommerce app. In a social app, it might be users sharing content. It s in this case that A/B testing an app is most similar to testing a website landing page, and identifying which visual features content changes improve conversion rate. However in apps you have the ability to test around events, or actions taken, making it more reactive to visitor flow and interaction than something like a landing page A/B test. A/B tests allow you to offer different versions of features and content -- buttons, icons, colors, screen flow and home screen layout, to name a few -- to different segments of your audience so they all experience the same app but in different ways. The results of various tests can show you which tested feature moves a user closer to or more quickly towards the ultimate conversion event -- a purchase, an ad click, or some other measure of engagement What Can You A/B Test?

14 4 TIPS AND TRICKS FOR TOP-NOTCH APP A/B TESTING

15 4 TIPS AND TRICKS FOR TOP-NOTCH APP A/B TESTING A/B testing offers a tremendous opportunity to use real data from real users to prove what works in your app. But doing the most successful A/B testing possible - that is, moving the needle on the metrics that matter most -- is easier said than done, especially if you re just getting started. Here are six essential elements to keep in mind as you dive into the mobile A/B testing waters: 1 Set clear goals that can be proven analytically A/B testing itself is just an orchestration of an experiment. What s important, though, are the implications and analysis of the test. So, consider what you want to analyze -- what repercussions will the test have for the app experience as a whole? What are you trying to learn? Having a set hypothesis that you know can be proven or disproven by the data is the only way to identify clear results and move forward. 2 Look for quick wins Until you can prove the power of A/B testing to your company s decision-makers, keep your tests fast and simple, focusing on quick wins. Look at easily achievable elements you can test, like push messaging content, to run simple but effective campaigns. This way, you can help your organization see the value of A/B testing as early as possible before taking a deeper data dive. 3 Test frequently and iteratively A/B testing of your mobile app is never one and done. Instead, it s an ongoing process of continuous learning, which means testing, testing and then more testing. The power of automated A/B testing is that you can have short test cycles where engineering doesn t have to be involved -- so you can take advantage of your testing platform to test regularly and repeatedly. And when you measure against control groups, you can dynamically change messaging and content in response, allowing for easier, smarter testing Tips and Tricks for Top-Notch App A/B Testing

16 4 Test by user segment Instead of just testing across all audiences broadly, you can use segmentation to create highly unique user groups, resulting in richer end results. Running multiple A/B tests across your different segments ensures that you ll be gaining the most targeted test results possible from users who naturally behave similarly. Plus, it allows you to conduct more complex testing across campaigns for better behavioral insight. 5 Test visitor flows Testing and tracking different visitor screen flows is unique to app analytics, and is key to determining fall-off points and conversions within A/B campaigns. For example, group A may be given only a one-screen conversion process, but group B has a two-screen process; one is faster but conveys less information, the other takes more time but provides added context. You can test which screen experiences drive the greatest impact on retention and revenue, in addition to testing messaging, context and visuals, for the most in-depth results. 6 Try multivariate testing There are many small features you can address with a simple A/B test. The next step to assessing deeper interactions is multivariate testing -- where you test two different elements at the same time, such as the order of a feature set as well as the color, particularly by segment for enhanced results. This needs to be built on a foundation of preliminary A/B tests that have shown clear and actionable insight. Moving to the multivariate level requires a baseline set of analytics that prove why you re testing the next round of features Tips and Tricks for Top-Notch App A/B Testing

17 5 A REAL-LIFE A/B TEST EXAMPLE: RUNKEEPER

18 5 REAL-LIFE CASE STUDY: RUNKEEPER Known as the personal trainer in your pocket, Localytics customer RunKeeper is the leading health and fitness app for ios and Android with more than 25,000,000 organically acquired users. As your digital trainer, RunKeeper uses the sensors built into phones to track your physical activity. The Challenge RunKeeper is a leading health and fitness app for ios and Android with more than 25 million organically- acquired users. Looking to move far beyond the running category, RunKeeper wanted to find ways to ignite enthusiasm and bring awareness to using the app to track other activities like weight lifting, yoga or biking. Localytics saves me time and makes it a lot easier to communicate test results. It provides actual insight. Localytics makes my life easier - Jon Gillman, Product Manager at RunKeeper The Solution RunKeeper ran an A/B test of their start screen using Localytics to see if a design change provoked users to log non-running activities. Specifically, they used funnel analysis tools to split the reports with custom dimensions for the A/B test groups and analyze which UI variation resulted in more non- running events logged A Real-Life A/B Test Example: RunKeeper

19 The Results The winning version was 10 times more successful at driving users to log a non-running event than the other version Since the A/B test was exclusively done in their Android app, RunKeeper was able to port the winning variation over to their iphone app with extra confidence saving development time on iterations later on A Real-Life A/B Test Example: RunKeeper

20 6 WINNING IN THE MOBILE MARKET

21 6 WINNING IN THE MOBILE MARKET The powerful one-two-punch of analytics and testing provides undeniable results that are hard for the higher-ups to ignore. Investment in mobile marketing is expected to grow by 38% over the next five years. With more brands crowding the app space, there is far more competition for users attention. In order to rise above the app noise, marketers need a strong foundation of data and insights to formulate an app strategy. Analytics give you the overall picture, but A/B testing takes assumptions and turns them into actionable facts. The powerful one-two-punch of analytics and testing provides undeniable results that are hard for the higher-ups to ignore. Whether you test in-app messages, push messages or app content and features, experimenting with ongoing, simple A/B tests on a variety of app elements can make the difference between an app that drives user engagement, retention and conversion and one that doesn t live up to its promise Winning in the Mobile Market

22 START YOUR FREE TRIAL Trusted by top brands to provide actionable insight. Start your free trial of Localytics today Localytics is the leading marketing and analytics platform for mobile and web apps. Localytics works with some of the world s most well-known brands, like ebay, Salesforce, Microsoft and The New York Times, helping them create great experiences for their customers and maximizing profitability, engagement, lifetime value, and loyalty. Localytics provides analytics and marketing for more than 1 billion devices across more than 20,000 apps. For more information visit or contact sales@localytics.com to request a demo.