RunKeeper's Applications Run Better with Applause's In-the-Wild Testing

Transcription

1 IDC ExpertROI SPOTLIGHT RunKeeper's Applications Run Better with Applause's In-the-Wild Testing Sponsored by: Applause Matthew Marden December 2014 Overview RunKeeper of Boston, Massachusetts, develops applications for iphone and Android smartphones that allow users to track their fitness activities. RunKeeper's success in continuing to grow its user base of 5 to 6 million active users per month depends on delivering user-friendly and reliable fitness trackers that offer more functionality with every release cycle. Founded in 2008, RunKeeper has built up a user base that is approaching 40 million between its two fitness trackers the RunKeeper fitness-tracking application and the Breeze daily steptracking application. Users can choose whether to use these trackers passively (i.e., capturing statistics about their activities) or more actively (i.e., creating detailed fitness plans and sharing personal content with friends). With awareness on the rise about the value of fitness trackers and wearables more generally, RunKeeper continues to grow its user base. As this happens, the company is moving into more non-u.s. markets that require localization of its applications. RunKeeper has been challenged to match its quality assurance (QA) efforts to its rapidly growing user base. Ben Bates, Head of Quality at RunKeeper, explained that he was brought on board in 2012 with the charge of prioritizing RunKeeper's QA efforts to support future growth. The QA team performs extensive regression testing on every application release, but Bates noted that identifying all possible issues prior to application release used to be a struggle with only internal staff resources. This situation led to crashes and costs associated with fixing bugs. Business Value Highlights Organization: RunKeeper Location: Boston, Massachusetts Challenge: Improve quality of applications by identifying and catching more bugs while promoting overall quality assurance efforts Solution: Applause's in-the-wild application testing solution Five-Year Cumulative Benefits: $377,600 ROI of 325% Payback in 4.5 months Other Benefits: 42% more application releases 25% more features per application Catching additional three bugs per application release 92% faster application localization verification Localized application versions 1.5 weeks faster to market on average In August 2013, RunKeeper decided to trial Applause's in-the-wild application testing services. 1 Bates explained, "We wanted to move fast, and what Applause was saying made sense. We began a trial with Applause and thought that we might as well see what they can do with it." 1 utest changed its name to Applause in spring December 2014, IDC #253316

2 Three months later, RunKeeper decided to use Applause on a subscription basis, after concluding that Applause was enabling it to get ahead of problems and improve the performance of its applications. With Applause's in-the-wild testing services, RunKeeper is now able to identify and handle more bugs before they cause crashes or other performance degradations. As a result, its trackers are more robust and perform better. This has helped RunKeeper continue to find new partners for sponsorships, events, and challenges and makes its users more likely to continue to use its applications and potentially become "for pay" users. Meanwhile, more robust testing with Applause means that RunKeeper releases new versions of its applications more frequently and on a more regular basis. In addition, RunKeeper has leveraged its use of Applause to ease the challenge of localizing applications as it moves into new markets, which has helped it get its applications to a number of non-englishspeaking markets several weeks earlier, pushing forward the timeline in which it can earn revenue. Based on interviews with Bates, IDC calculates that RunKeeper will achieve benefits worth a total of $377,600 over five years through its use of Applause's in-the-wild testing services. This means that RunKeeper will realize a return on investment (ROI) of 325% and a breakeven on its investment in Applause in 4.5 months. Implementation Applause first approached RunKeeper about its in-the-wild testing services in At that time, RunKeeper chose to focus on building its internal QA team's capabilities before considering in-the-wild testing services. When Applause approached RunKeeper again in mid-2013, Bates believed that it made sense for his team to give in-the-wild testing services a chance. He explained, "We thought we would see what Applause can do with our applications. So we gave them free range and told them to find all of the bugs they can." Bates elaborated, "Regression testing at the end of our testing cycles takes a long time and there's a human element because if we don't have all of the pieces out there, then we might miss something with only several people testing. It made sense for us to get these tests into the hands of Applause and have more people who will spend the time testing." "With Applause, we are able to get ahead of a lot of the crashes we were concerned about and not wait until the applications were already in the store. We are finding more bugs with Applause prior to release that we weren't necessarily finding before." After a three-month trial, RunKeeper moved to a full subscription model with Applause. "With Applause, we are able to get ahead of a lot of the crashes we were concerned about and not wait until the applications were already in the store," Bates said. "We are finding more bugs with Applause prior to release that we weren't necessarily finding before." RunKeeper can now use Applause to run tests on two applications at once. According to Bates, RunKeeper now usually has ongoing tests with Applause for the iphone and Android versions of the RunKeeper application, and it has Applause test its Breeze application on an as-needed basis. Bates said that Applause typically puts about 20 testers on each RunKeeper application that it tests, after RunKeeper uploads the beta version of an application release to Applause and provides it with an overview of the types of issues that it wants to focus on when conducting final testing IDC #

3 RunKeeper is also leveraging Applause's worldwide community of testers for localization. It has already used Applause to conduct localized testing in 12 languages by being able to request testers who speak certain languages. Bates said that he expects RunKeeper to continue to leverage Applause's ability to get RunKeeper applications on the devices of local language speakers as the company expands to new markets. Finally, Applause Analytics is one of several tools RunKeeper leverages to capture more insights about the performance of its applications, and the company continues to explore use cases for it. Benefits Since beginning to use Applause's in-the-wild testing services in September 2013, RunKeeper has improved the robustness and consistency of its QA efforts. This results in better-performing applications and enables RunKeeper to release more applications with the same size QA staff while adding more features with additional functionality to each application release. RunKeeper has benefited from its use of Applause's in-the-wild testing services by introducing a more regular cadence to its application release cycle. Its QA team knows that it can rely on Applause to augment its regression testing efforts, so it has the confidence needed to go live with application releases sooner. With Applause, RunKeeper is able to maintain a release schedule of approximately once every five weeks for its iphone and Android RunKeeper applications and carry out releases of its Breeze application as needed. For RunKeeper, the consistency and the frequency of release are improvements on the largely ad hoc release process it followed before using Applause. Timely application releases with new functionality are at the core of RunKeeper's value proposition, underscoring Applause's impact on operations. "With Applause, we are able to move more quickly in general with our testing and carry this through to our application release cycles. Not only do we release new features faster, but we also have more confidence in our product." Thanks to the testing carried out by Applause's team of testers, RunKeeper has been able to issue more application releases and add more new features without having to increase the size of its QA staff commensurately. Bates noted, "With Applause, we are able to move more quickly in general with our testing and carry this through to our application release cycles. Not only do we release new features faster, but we also have more confidence in our product." Applause has also helped RunKeeper improve the quality of its applications. According to Bates, Applause helps RunKeeper identify an average of almost three more bugs per application release cycle that it is able to fix prior to the application's release. This saves RunKeeper's QA team the time previously spent fixing these bugs when they were discovered post-release and also means that users experience fewer crashes and problems with RunKeeper's trackers. RunKeeper's ability to catch and fix more bugs before applications are released leads to higher satisfaction levels among the company's users and partners. Users expect RunKeeper's trackers to work and take into account the reliability of the product when deciding whether to continue to use it or even possibly upgrade to "for pay" services. In addition, RunKeeper's business model depends on developing and maintaining partners for events and challenges, and the reliability of RunKeeper's 2014 IDC #

4 applications is critical to the company's efforts to build its business and partnerships. Thus, by improving the reliability of its tracker applications with Applause, RunKeeper is boosting its bottom line and helping grow its business. Bates also credits Applause with supporting his company's growth efforts. Importantly, RunKeeper is leveraging Applause to make its localization efforts for new markets more efficient. RunKeeper has already localized its trackers for 12 non-u.s. markets with Applause. With Applause in place, RunKeeper has reduced the time needed to verify application localization by 92% and can release applications to these markets an average of 1 2 weeks faster. This means that RunKeeper is able to capture revenue from applications in these markets earlier than it otherwise would have, although IDC has not quantified this revenue benefit for the purposes of this study. Bates said that as RunKeeper continues to grow, it will continue to use Applause to make sure that its applications appeal to its growing user base around the world. With Applause in place, RunKeeper has reduced the time needed to verify application localization by 92% and can get its applications to these markets an average of 1 2 weeks faster. Quantifying the Benefits IDC has quantified the benefits RunKeeper is achieving with its use of Applause in-the-wild testing services based on several interviews with Bates. IDC calculates that over five years, RunKeeper will realize benefits worth an average of $105,600 per year through QA staff efficiencies and reduced staff time needed to repair bugs and verify application localization. The most significant benefit to RunKeeper from its use of Applause in-the-wild testing services has been its ability to issue more application releases with more features with the same QA staff. In its first year of using Applause, RunKeeper carried out 42% more application releases and added 25% more features per release without increasing the size of its QA team. This reflects a significant productivity leap for RunKeeper's QA team, and the speed and quality of application releases are at the heart of RunKeeper's business development efforts. In addition, Applause saves time for RunKeeper's QA employees because they have to repair fewer bugs after applications are released. According to Bates, because Applause adds a layer of targeted testing that uses applications in locations and situations that the QA team could not otherwise cover, RunKeeper benefits from the identification of three bugs per application release that it can fix before an application is released. When these bugs are not identified before release, it takes the QA team about six hours of employee time in total, which adds up to a substantial amount of time given that RunKeeper is now at about 25 releases per year. Finally, RunKeeper has made its application localization efforts more efficient with Applause. Its QA team now spends less time trying to verify the quality of application localizations and has more confidence in the performance of localized applications. This saves just under three hours of employee time per localized application release and provides RunKeeper's dedicated localization project manager with efficiencies that increase her productivity IDC #

5 Return on Investment IDC calculates that RunKeeper will capture discounted benefits worth $377,600 over five years through its use of Applause in-the-wild testing services compared with a discounted investment of $88,800. As a result, RunKeeper is set to achieve a five-year ROI of 325% with breakeven on its investment in Applause occurring in 4.5 months (see Table 1). TABLE 1 Five-Year ROI Analysis Benefit (discounted) $377,600 Investment (discounted) $88,800 Net present value (NPV) $288,800 Return on investment (ROI) 325% Payback period 4.5 months Discount rate 12% Source: IDC, 2014 IDC conducted several interviews with RunKeeper to quantify the benefits and investment associated with its use of Applause in-the-wild testing services and created an ROI analysis from the results. IDC calculates the ROI and payback period in a three-step process: 1. Measure the benefits from increased IT staff and user productivity and other cost savings since deployment. 2. Ascertain the total investment. 3. Project the investment and benefit over five years and calculate the ROI and payback period. The ROI is the five-year net present value of the benefit divided by the discounted investment. To account for the time value of money, IDC bases the ROI and payback period calculations on a 12% discounted cash flow IDC #

6 About IDC International Data Corporation (IDC) is the premier global provider of market intelligence, advisory services, and events for the information technology, telecommunications and consumer technology markets. IDC helps IT professionals, business executives, and the investment community make factbased decisions on technology purchases and business strategy. More than 1,100 IDC analysts provide global, regional, and local expertise on technology and industry opportunities and trends in over 110 countries worldwide. For 50 years, IDC has provided strategic insights to help our clients achieve their key business objectives. IDC is a subsidiary of IDG, the world's leading technology media, research, and events company. Global Headquarters 5 Speen Street Framingham, MA USA idc-insights-community.com Copyright Notice External Publication of IDC Information and Data Any IDC information that is to be used in advertising, press releases, or promotional materials requires prior written approval from the appropriate IDC Vice President or Country Manager. A draft of the proposed document should accompany any such request. IDC reserves the right to deny approval of external usage for any reason. Copyright 2014 IDC. Reproduction without written permission is completely forbidden.