2017 MOBILE TESTING STRATEGY Ensure Mobile App Success in 7 Steps
Table of Contents INTRODUCTION 3 USER EXPERIENCE 4 DATA INTEGRITY 5 BE AGILE AND TEST YOUR APP 6 7 STEPS TO A SOUND MOBILE TESTING STRATEGY 7 ADDRESS DEVICE AND OS FRAGMENTATION 8 MANAGE SIZE AND CHATTY-NESS OF YOUR APP 9 TEST EARLY, TEST OFTEN 10 TEST ALONG THE PROCESS 11 USE MOBILE ANALYTICS 12 TEST YOUR ANALYTICS IMPLEMENTATION 13 STRATEGY START WITH THE END IN MIND 14 CONCLUSION 15 ABOUT THE AUTHOR 16
INTRODUCTION Agile product development with rapid release cycles allows companies to make their mobile products dynamic. The product becomes an experience in and of itself, establishing a feedback cycle with the end user instead of being static. While agile development has become the norm 90% of organizations practice agile it has presented somewhat of a paradox for mobile product developers. Agile dev makes for rapid iteration, continuously refining the experience based on robust data gathered from user behavior and customer feedback. The issue, however, is that agility sometimes comes at the expense of quality, specifically in two areas: user experience and data integrity. 3
USER EXPERIENCE A high-quality user experience is the product of a combination of beautiful design, personalization and seamless, intuitive user flows. While agile developers are able to quickly build and change mobile app features to accommodate the evolving needs and wants of their users, this agility can often result in tunnel-vision when it comes to mobile app testing. What can end up happening is, in an effort to quickly vet the functionality of a new feature before moving on to the next scheduled release, developers fail to take into account the effect new features will have on pre-existing features of the product. This passing-over of the testing phase usually isn t done out of ignorance, but rather out of duress: short release cycles make it difficult to set aside sufficient time to properly test each and every use case that could potentially break your mobile app. The ultimate result is developers finding themselves facing serious challenges further down the line as unaddressed issues rapidly accrue interest on complexity debt with each additional release. 4
DATA INTEGRITY Analytics data is both the end result and the catalyst of mobile app development. Data-driven DevOps teams collect and surface data in order to understand user behavior and how to optimize app performance. Data becomes the lifeblood of DevOps strategy, and thus accurate data is a key objective. With mobile SDKs making it relatively easy to implement analytics on your mobile app, mobile developers may fall into the trap of thinking they re golden, with no need to validate the functionality of their analytics SDK. Unfortunately, if not properly validated, developers and product teams may fail to realize that not every screen is tagged, or that the analytics tool is not collecting the correct data or collecting the data correctly. Unlike a broken screen flow, errors in analytics data may not be as visibly obvious. You may not notice something is amiss until your data starts looking awry, but by then you ve already lost valuable data that you have no way of recovering. Plus, if you re not keeping tabs on how your SDKs are affecting mobile app performance, it could result in a negative user experience. 5
BE AGILE AND TEST YOUR APP If you re not careful, agile development without a proper testing strategy could result in exactly the problems you were hoping to avoid when going agile: excessive development complexity, negative user experience and long time-to-market. Still, to remain agile you don t have time to manually replicate thousands of quality assurance tests on each new iteration of your mobile app. Consequently, you need to trust in automated mobile app testing tools where you can continuously build a portfolio of use cases to test, all of which can be automatically run on each new iteration of your mobile app. 6
7 Steps TO A SOUND MOBILE TESTING STRATEGY Strategy should always come before action. Building a mobile testing strategy during the early phases of development will ensure your app stays clean and effective for users. Consider the following steps when developing your mobile testing strategy: 1 Address device and OS fragmentation 2 Manage size and chatty-ness of your app 3 Test early, test often 4 Test along the process 5 Use mobile analytics 6 Test your analytics implementation 7 Strategy Start with the end in mind 7
1 ADDRESS DEVICE AND OS FRAGMENTATION The mobile market is divided across devices and platforms, and so is mobile app development. The two principal platforms dominate the market 99.6 percent of new smartphones run either Android or ios. Because both platforms hold a large share of the mobile market, most mobile-friendly organizations will build two separate apps, often created by two different teams. If proper analytics strategy isn t developed and applied, disparate teams may create disparate definitions of conversion events and use different naming conventions. To prevent disparate data, decide on unique user identifiers, conversion events and naming conventions for analytics variables that are independent of operating system. Share code as much as possible using IDEs like Xamarin (think hybrid app instead of two separate native apps). 8
2 MANAGE THE SIZE AND CHATTY-NESS OF YOUR APP Negative user experiences due to overly bulky mobile apps, or apps that spin up the network radio frequently, will result in rapid uninstalls and negative reviews. 50% of mobile users uninstall apps because they take up too much storage. While mobile SDKs can extend the features of your app and are relatively easy to implement, too many is just too many. Be sure to monitor network traffic, look for ways to batch requests, and use long polling, SSE or websockets to minimize data transfer to keep your app performant and lean. 9
3 TEST EARLY, TEST OFTEN Similar to the release early, release often mantra of agile developers, a test early, test often mentality will ensure you resolve issues earlier rather than later (which is must less costly). The sooner you identify problems, the sooner you can resolve them. Early development requires you to build a minimum viable product (MVP), and this will be key to ensuring your first-party code and third-party SDKs are working in tandem without any issues. Using a combination of code spot-checking and automated testing will allow you to identify points of failure during the initial development phases. 10
4 TEST ALONG THE PROCESS There are myriad steps that go into developing, testing and iterating a successful mobile application. Integrating testing into each step of the process is agility insurance, protecting you against complexity debt. UNIT TESTING Unit testing is the process of testing individual components of your code base, independent of other components. This allows you to isolate issues as you are adding new features to your technology. ALPHA/BETA TESTING Testing the usability of your mobile app with real end users will help ensure you re not only resolving technical issues, but also meeting the needs of your target user base. Internal feedback can be very helpful, but if you can organize a customer advisory board to perform alpha testing on an early prototype, you ll get much more relevant insight on your minimum viable product. When you ve resolved key concerns surfaced during alpha testing, you ll be able to move on to beta testing, where users can expose unique use cases, usability issues and potential points of failure. 11
5 USE MOBILE ANALYTICS Analytics can be considered a form of testing in that the data collected by analytics tools can help you identify bottlenecks, bugs and use cases that cause crashes. There are different categories of mobile analytics to consider, including: 1 In-app analytics 2 A/B testing 3 Crash reporting Voice of Customer analytics 5 Ratings 6 7 Push analytics 4 App store analytics As a special note, consider the roles of real-time data vs. historical data: Real-time data immediately transmits data based on behavioral events defined in your analytics variables, and allows you to respond immediately. Real-time data is important for location- and event-specific marketing, allowing you to reach a customer at the opportune moment. There are some implications to be considered with real-time data, however. You will need to be prepared to process data rapidly. You will need to have some sort of automation or AI set up to respond to real-time data in order to get the true value from this data. Historical data, on the other hand, is less time-sensitive, thus allowing you to focus less on immediate action and more on deeper analysis. The way in which you intend to consume the data will determine which style of implementation you will need. 12
6 TEST YOUR MOBILE ANALYTICS IMPLEMENTATION Just like any other component of your mobile application, your analytics implementation can break. And if it breaks, you lose key data surrounding user behavior for as long as you remain unaware of the failure, plus the amount of time it takes you to identify what has broken and why. The later you identify the issue, the higher the cost of remedying the challenge. Finding an issue in production costs 100x more than identifying the problem during the development phase. (Source: isixsigma) Spot-checking your app code to ensure everything is implemented correctly is one method to ensure your analytics code is working properly, but this sort of quality assurance is not scalable and is only a temporary solution. 13
7 STRATEGY START WITH THE END IN MIND Involving the right people with the correct documentation from the very beginning will ensure you accomplish your strategy. It is best practice to establish a data governance council that is comprised of all data stakeholders. This council is headed by a chief data steward, who is responsible for ensuring the council meets frequently (monthly or quarterly) and that all analytics documentation is up to date. An analytics strategy that goes undocumented is not much of a strategy. Having your KPIs, metrics and goals written out, along with clear and dynamic variable strategy documentation (sometimes called a solution design reference), will ensure your company has a true north for your testing compass to navigate towards. 14
CONCLUSION As stated before, mobile app testing is agility insurance. If you want both your app and your development processes to remain dynamic, minimizing the number of bugs that slip through to a production environment will greatly increase the probability of success in an incredibly competitive mobile app market. Introducing automation into your mobile testing strategy can make all the difference for an overwhelmed development team or a team hoping to speed up release cycles by isolating broken code without having to manually comb through thousands of lines of code. AppAssurance is ObservePoint s patented technology for scanning mobile apps, capturing data SDK functionality, and validating your implementation against pre-defined compliance rules. Request a free sample audit of your app to see how automation can streamline testing processes. 15
ABOUT THE AUTHOR Sun is currently Director of Product Management at ObservePoint. She is passionate about internet products and marketing. Sun conceptualizes and drives change in an impactful and sustainable way. Before becoming Director of PM, Sun oversaw product innovation for AppAssurance, ObservePoint s mobile app tag and data quality platform. In past roles, Sun has contributed to the product innovation of Deutsche Telekom, T-Online International AG, and Fast Multimedia AG. SUN SNEED