The Total Economic Impact Of SiteSpect Origin Experiments

Transcription

1 A Forrester Total Economic Impact Study Commissioned By SiteSpect Project Director: Sean McCormick May 2016 The Total Economic Impact Of SiteSpect Origin Experiments Cost Savings And Business Benefits Enabled By Origin Experiments

2 Table Of Contents Executive Summary... 3 Disclosures... 5 TEI Framework And Methodology... 6 Analysis... 7 Financial Summary SiteSpect Origin Experiments: Overview Appendix A: Total Economic Impact Overview Appendix B: Forrester And The Age Of The Customer Appendix C: Glossary Appendix D: Supplemental Material Appendix E: Endnotes ABOUT FORRESTER CONSULTING Forrester Consulting provides independent and objective research-based consulting to help leaders succeed in their organizations. Ranging in scope from a short strategy session to custom projects, Forrester s Consulting services connect you directly with research analysts who apply expert insight to your specific business challenges. For more information, visit forrester.com/consulting. 2016, Forrester Research, Inc. All rights reserved. Unauthorized reproduction is strictly prohibited. Information is based on best available resources. Opinions reflect judgment at the time and are subject to change. Forrester, Technographics, Forrester Wave, RoleView, TechRadar, and Total Economic Impact are trademarks of Forrester Research, Inc. All other trademarks are the property of their respective companies. For additional information, go to

3 3 Executive Summary SiteSpect commissioned Forrester Consulting to conduct a Total Economic Impact (TEI) study and examine the potential benefits and costs enterprises may experience by deploying Origin Experiments, a core capability of the SiteSpect solution. The purpose of this study is to provide readers with a framework to evaluate the potential financial impact of SiteSpect Origin Experiments on their organizations, by leveraging online A/B and multivariate testing technology for continuous optimization in order to win, serve, and retain customers. SiteSpect Origin Experiments reduces the risk of deploying updates adversely affecting KPIs and helps identify value-generating changes. The costs and benefits for an ecommerce organization with over $1 billion in online revenue, based on a customer interview, are: Annual subscription costs: $91,350. Annual testing costs: $423,125. SiteSpect is an online testing and optimization solution. SiteSpect s Origin Experiments capability allows clients to quantify the impact of any back-end site change, enabling Annual savings and benefits: $13,324,552. product teams to measure, analyze, and optimize functionality. Clients can optimize functionality such as feature launches, search algorithms, checkout flow, business logic, and pricing strategies while delivering the optimal experience for each customer segment or even to specific users. SiteSpect Origin Experiments also enables the team to quantify the impact of functional changes to the digital experience, providing a return on investment (ROI) model that links development efforts to business KPIs such as conversion rate, average order value, retention and loyalty, and so on. To better understand the benefits, costs, and risks associated with SiteSpect Origin Experiments functionality, Forrester interviewed an existing customer with multiple years of experience using Origin Experiments. Prior to SiteSpect Origin Experiments, the customer used a rudimentary homegrown cookie-based testing solution that was poorly utilized. The lack of an adequate testing solution led to customers experiencing many issues as new releases were pushed out; created a longer and inflexible release process; and prevented developers from moving onto the next release, as they were busy identifying and fixing issues as they arose. With SiteSpect Origin Experiments, the customer no longer has to rely on a push and pray model, where it hopes that new functionality and features, as well as back-end site improvements, will generate positive results. The customer is now able to test for positive results on a much smaller percentage of its traffic and move quicker on scaling these improvements to all its customers. Additionally, as the organization became more sophisticated in its Agile development process, SiteSpect Origin Experiments helped to enable a more continuous release cycle versus the handful of releases per year the organization had been bound to. Said one lead data scientist, It took a while before we realized Oh, we could test that before you just turn it on, and now as we ve become more sophisticated with a data science team, we ve been able to use it for algorithm testing, leading to a better customer experience. SITESPECT REDUCES RISK IN DEPLOYMENT AND GENERATES NEW SALES Our interviews with an existing customer and subsequent financial analysis incorporating results from a collection of SiteSpect customers resulted in the risk-adjusted benefits and costs shown in Figure 1. 1 This analysis points to benefits of $13.3 million per year versus implementation and ongoing costs of $1.3 million over three years, adding up to a net present value (NPV) of $38.7 million and a 30 to 1 return on investment. This translates to benefits of more than $6.2 million per year in reducing the risk of implementing updates that adversely affect KPIs. With Origin Experiments, on average, a 3% decrease in average order value (AOV) was prevented in 22.5% of tests. Additional benefits were realized in testing for value-generating updates that were then able to be scaled out across all site traffic.

4 4 FIGURE 1 Financial Summary Showing Three-Year Risk-Adjusted Results Total threeyear benefits: $39,973,657 Return on investment: 30x Average order value risk reduction: 3.0% Average order value testing improvements: 10.5% Benefits. The interviewed organization experienced the following risk-adjusted benefits: Testing risk reduction helped improve annual profits by $5 million to $10 million per year. By identifying average order value decreases in updates before they are pushed into full production, SiteSpect Origin Experiments was able to avoid an average of 2% to 4% AOV decreases in 22.5% of tests. This was particularly helpful in optimizing add-to-cart functionality. Testing improvements helped improve annual profits by $7 million to $10 million per year. By identifying those updates that maximize average order value, SiteSpect Origin Experiments was able to improve an average of 8.5% to 12.5% AOV increases in 7.5% of tests. One example from the interviewed organization was in geotargeting promotions. It was able to identify AOV increases in specific promotions in specific geographies, helping to optimize for the right promotions in each market. Labor efficiencies in replatforming site search technology of $387,000 in cost savings. The interviewed organization was able to speed up the development process by over 5,000 hours during its search replatform initiative by developing algorithms and then testing the performance of those algorithms to ensure there wasn t a negative impact on the customer experience or decreases in average order value. These algorithms allowed the organization to avoid redeveloping all the rules and configurations from its legacy platform. Costs. The interviewed organization experienced the following three-year present value risk-adjusted costs: Software subscription costs of $227,174 over three years, or $91,350 per year. These are monthly fees paid to SiteSpect for access to Origin Experiments. Professional services fees of $137,088 over three years, or $750 per test on average. The organization pays SiteSpect professional services fees to help set up and analyze tests. Internal testing and analysis labor costs of $922,419 over three years. A team of eight data scientists, testers, and developers dedicate approximately 35% to 45% of their time to creating, running, and analyzing SiteSpect Origin Experiments tests.

5 5 Disclosures The reader should be aware of the following: The study is commissioned by SiteSpect and delivered by Forrester Consulting. It is not meant to be used as a competitive analysis. Forrester makes no assumptions as to the potential ROI that other organizations will receive. Forrester strongly advises that readers use their own estimates within the framework provided in the report to determine the appropriateness of an investment in SiteSpect Origin Experiments. SiteSpect reviewed and provided feedback to Forrester, but Forrester maintains editorial control over the study and its findings and does not accept changes to the study that contradict Forrester's findings or obscure the meaning of the study. SiteSpect provided the customer for the interview but did not participate in the interviews.

6 6 TEI Framework And Methodology INTRODUCTION From the information provided in the interviews, Forrester has constructed a Total Economic Impact (TEI) framework for those organizations considering implementing SiteSpect/Origin Experiments. The objective of the framework is to identify the cost, benefit, flexibility, and risk factors that affect the investment decision, to help organizations understand how to take advantage of specific benefits, reduce costs, and improve the overall business goals of winning, serving, and retaining customers. APPROACH AND METHODOLOGY Forrester took a multistep approach to evaluate the impact that SiteSpect Origin Experiments can have on an organization (see Figure 2). Specifically, we: Interviewed SiteSpect product and consulting personnel, along with Forrester analysts, to gather data relative to SiteSpect Origin Experiments and the marketplace for Origin Experiments. Interviewed one organization currently using SiteSpect Origin Experiments to obtain data with respect to costs, benefits, and risks. Analyzed results for a collection of customers that use SiteSpect Origin Experiments to determine the aggregate effect. Constructed a financial model representative of the interview using the TEI methodology. The financial model is populated with the cost and benefit data obtained from the interviews. Risk-adjusted the financial model based on issues and concerns the interviewed organization highlighted in interviews. Risk adjustment is a key part of the TEI methodology. While the interviewed organization provided cost and benefit estimates, some categories included a broad range of responses or had a number of outside forces that might have affected the results. For that reason, some cost and benefit totals have been risk-adjusted and are detailed in each relevant section. Forrester employed four fundamental elements of TEI in modeling the SiteSpect Origin Experiments solution: benefits, costs, flexibility, and risks. Given the increasing sophistication that enterprises have regarding ROI analyses related to IT investments, Forrester s TEI methodology serves to provide a complete picture of the total economic impact of purchase decisions. Please see Appendix A for additional information on the TEI methodology. FIGURE 2 TEI Approach Perform due diligence Conduct customer interviews Construct financial model using TEI framework Write case study

7 7 Analysis ORGANIZATION OVERVIEW For this study, Forrester conducted an interview with a representative from a data-driven company, which is a SiteSpect customer based in North America: A retail organization with over 500 stores within North America and an ecommerce site that generates more than US$1 billion in annual revenue. The ecommerce team utilizes an Agile development methodology with continuous release cycles and is structured with several product teams that are supported by data scientists. Based on the interviews and the results of a collection of customers provided by SiteSpect, Forrester constructed a TEI framework and an associated ROI analysis that illustrates the areas financially affected. INTERVIEW HIGHLIGHTS In the age of the customer, it s imperative that organizations invest in transforming the digital customer experience. More than ever, customers are going online to shop and expect that their favorite retail stores will be there providing the same level of service and quality that they received in-store. However, as the power shifts to the customer, professionals responsible for customer experience, such as marketers, ecommerce pros, and product managers, must continuously work to evolve their digital touchpoints to optimize these customer experiences. Some organizations are applying analytics-driven approaches to their continuous optimization strategies in order to optimize the customer experience across these digital touchpoints. Online A/B and multivariate testing solutions such as SiteSpect are equipping these professionals with the data and insights to improve their customer experience, conversion rates, and revenue. By utilizing experimental design and statistical techniques that compare variations of a digital customer s interactions across the site, SiteSpect Origin Experiments optimizes functionality such as feature launches, search algorithms, checkout flow, business logic, and pricing strategies, while helping to deliver an optimal experience for customers. 2 After releasing we were able to identify an issue, isolate the problem, and fix within one day. SiteSpect Origin Experiments allowed us to push it back out slowly and ramp up, as everything looked good. This was the first time we didn t have to rely on the push and pray model and could be confident in the results. ~ Lead data scientist, Fortune 100 retailer Prior to implementing SiteSpect and using Origin Experiments, the interviewed organization was far less sophisticated in its digital business, many times making feature, design, and back-end functional decisions based on expectations and previous expertise rather than data-driven decisions, which led to suboptimal customer experiences. As the organization became more sophisticated in its digital business, embracing Agile development practices, organizing product teams, and building and adding data scientist roles, the need arose for a sophisticated online testing solution to help optimize its digital experience and directly improve revenue. SiteSpect Origin Experiments provides digital teams with actionable insights in a timely manner, removing the old push and pray model. In the past, teams would hypothesize and predict results but not have the ability to quantify the impact. SiteSpect Origin Experiments enables companies to evolve into a data-driven continuous release process. Using SiteSpect Origin Experiments also helps mitigate the impact of errors in releases and speed up the feedback loop and rework process. Additional capabilities provided by SiteSpect Origin Experiments include:

8 8 Algorithm testing, giving organizations the ability to optimize search, recommendations, and profiling models. Release testing to mitigate risks in new code that negatively affect already established code or functionality. Feature testing to improve current features or processes like checkout flow or add-to-cart capability to provide a better digital experience. New feature testing, allowing organizations to create new functionality and ensure a positive customer reaction. I think it s [SiteSpect] essential to our Agile development methodology. Without it, it just seems like insanity ~ Lead data scientist, Fortune 100 retailer The interview revealed the following improvements. SiteSpect Origin Experiments helped the interviewed organization to: Streamline online search replatforming while ensuring performance was as good or better. One of the most significant benefits experienced by the interviewed organization was during its site search replatform effort. While its previous search solution was developed and refined over many years, including a large amount of hours spent building customizations and rules to improve performance, the organization hoped to avoid having to rebuild all the logic in the new platform by creating algorithms that could produce similar or better results. With SiteSpect Origin Experiments, these algorithms could now be tested and quickly iterated and refined to ensure that the performance was as good or better than the previous solution s custom rules and configurations. This not only helped speed up the time-to-market of the new search platform, but it saved hundreds of thousands of dollars in development. The interviewee explained, I would say the process would have taken eight to 12 months longer because we would have to make a rule, see what had changed, then repeat the process over and over again until we satisfied the requirements. In addition, SiteSpect Origin Experiments mitigated the risk of turning on new search algorithms to 100% of traffic before knowing the algorithms level of performance. This prevented customers from having bad experiences and being turned away by the new technology. Optimize in-store pickup and checkout features. SiteSpect Origin Experiments helps to test and optimize features and functionality on the web and mobile sites. One example communicated by the interviewed organization includes optimizing the in-store pickup feature. The organization found that increases with in-store pickup drove a reduction in online shipments, so as more customers chose in-store pickup, fewer shipments were required and vice versa. The interviewed organization felt there was an optimal balance between the amount of orders that would be shipped online and the amount of orders that would be picked up in-store. However, when the organization was building improvements to these features, it would over-index one way or the other. SiteSpect Origin Experiments gave the interviewed organization the ability to test before releasing feature changes that over-indexed, helping to optimize the features and maximize value. Transition from multipage to single-page checkout on the mobile site. SiteSpect Origin Experiments was used to test how customers would react to this change and how the single-page checkout could be optimized to reduce abandoned carts during checkout. The interviewed organization said that as a result, We saw definite improvements in completed checkouts and have continued utilizing the single-page checkout strategy. We saw definite improvements in completed checkouts and have continued utilizing the single-page checkout strategy. ~ Lead data scientist, Fortune 100 retailer

9 9 Mitigate risks in updating account sign-on. The interviewed organization noted the ability to mitigate risk in releases by testing the release on a smaller percentage of traffic before scaling to 100%. One example where significant issues were identified was in an account sign-on update. As the release went out, customers could no longer log into their accounts. This issue was quickly identified by SiteSpect Origin Experiments, whereby a 30% subset of traffic saw a 10% drop in conversion. Had this organization still relied on the push and pray model of its past, it could have lost significant revenue and seen a high degree of frustration from its most valuable group of customers. Use geotargeting promotions to increase effectiveness in specific markets. Another way the interviewed organization was utilizing SiteSpect Origin Experiments was in targeting promotions and testing their effectiveness in specific geographical markets. Through these tests, the interviewed organization was able to improve the effectiveness of promotions, leading to increased sales, increased sell-through of slow-moving inventory, and overall higher margins. In addition, as these promotions were found successful in one market, they could be applied to other markets that shared similar attributes. SiteSpect Origin Experiments even afforded the interviewed organization the ability to limit the liability of a promotion going viral by controlling the percentage of traffic the promotions are released to. This helped protect against large margin losses.

10 10 BENEFITS The interviewed organization experienced a number of quantified benefits in this case study: Testing risk reduction. Testing improvements. Labor avoidance in search replatform. Another important benefit mentioned by the interviewed organization was the ability to quantify the ROI of each individual development effort using business KPIs such as conversion and average order value. This allowed it to identify adverse effects and turn off code changes quickly. Previously, if a site feature change was launched that hindered a business KPI, the organization wouldn t know about it until days later. This meant developers had already begun working on the next project when the code would urgently need to be remediated. This created inefficiencies in the process and a more hectic and stressful environment for developers. With SiteSpect Origin Experiments, teams are able to identify negative code impacts very quickly and turn them off. Returning the code to the developers before they start on the next project allowed them to focus on fixing the issue at hand. As stated by the interviewee: Honestly, one of our biggest benefits with SiteSpect Origin Experiments is to be able to fix an issue when traffic is not flowing there. The issues get fixed much faster without impacting the overall customer experience during this process. Testing Risk Reduction The interviewed organization indicated that a key benefit from the SiteSpect Origin Experiments implementation was the ability to test for negative impacts online. Prior to SiteSpect Origin Experiments, the interviewed organization would have to push changes out into production and hope for the best, many times never truly understanding the drivers of negative performance in releases. Using Origin Experiments, the interviewed organization was able to optimize its ecommerce business, maximize income, and generate a better customer experience. With Origin Experiments, the interviewed organization ran an average of six tests per month, or 70 tests per year. Seventy-five percent of those tests were focused on risk reduction and 30% of those tests found statistically significant results, meaning roughly 16 of the 70 tests identified opportunities for improvement. Within these 16 tests, a 4% decrease in average order value was avoided in the first year. As the interviewed organization s ecommerce environment became more optimized, the decrease in average negative order value avoided reduced to 2%. With 3.5 million visitors per test and a 2.58% conversion rate, incorporating an AOV of $75 and assuming a 25% gross margin rate, the interviewed organization was able to save between $33,863 and $67,725 per test. Annualizing this benefit by taking the 365 days per year divided by the average test duration of 18 days, and then multiplying by 50% to account for testing during the second half of the year, the total annual benefit experienced by the interviewed organization was between $5.4 million and $10.8 million. Over the three-year period, the total testing risk reduction benefit equaled a present value of $20,597,753. The interviewed organization validated a range of variables relating to risk reduction. While averages from these ranges were utilized to calculate the testing risk reduction benefit, it was necessary to account for the variability within those ranges; therefore, this benefit was risk-adjusted and reduced by 10%. The risk-adjusted total benefit resulting from testing risk reduction over the three years was $18,537,978, or about $1.2 million per statistically significant risk reduction test.

11 11 TABLE 1 Testing Risk Reduction Ref. Metric Calculation Initial Year 1 Year 2 Year 3 A1 Average unique visitors per test 3,500,000 3,500,000 3,500,000 A2 Average conversion rate per test 2.58% 2.58% 2.58% A3 Average order value per test $75 $75 $75 A4 A5 Average negative order value decrease avoided per test Average testing revenue improvements per test 4% 3% 2% A1*A2*A3*A4 $270,900 $203,175 $135,450 A6 Average gross margin Assumption 25% 25% 25% A7 Average margin increase per test A5*A6 $67,725 $50,794 $33,863 A8 Tests per year A9 A10 Percent of total tests with statistically significant results Percent of statistically significant tests with risk avoidance 30% 30% 30% 75% 75% 75% A11 Average test duration A12 Annualization multiplier 365 / A11 * 50% At Testing risk reduction A7*A8*A9*A10*A12 $0 $10,814,836 $8,111,127 $5,407,418 Risk adjustment 10% Atr Testing risk reduction (riskadjusted) $0 $9,733,352 $7,300,014 $4,866,676 Testing Improvements Another key benefit indicated by the interviewed organization was the ability to test for positive impacts online. Prior to SiteSpect Origin Experiments, the interviewed organization would have to hypothesize features and functionality changes and then push and pray and hope for the best, many times never truly understanding if those changes drove improvements. Using SiteSpect Origin Experiments, the interviewed organization was able to improve its ecommerce business, maximize income, and deliver a better customer experience. With SiteSpect Origin Experiments, the interviewed organization ran an average of six tests per month, or 70 tests per year. Twenty-five percent of those tests were focused on analyzing function, feature, and release improvements and 30% of those tests found statistically significant results, meaning roughly five of the 70 tests identified significant improvements. Within these five tests, a 12.5% increase in average order value was identified in the first year. As the interviewed organization s ecommerce environment became more optimized, the increase in average order value decreased to 8.5%. With 3.5 million visitors per test and a 2.58% conversion

12 12 rate, incorporating an AOV of $75 and assuming a 25% gross margin rate, the interviewed organization was able to increase its margin between $143,916 and $211,641 per test. Annualizing this benefit by taking the 365 days per year divided by the average test duration of 18 days, and then multiplying by 50% to account for testing during the second half of the year, the total annual benefit experienced by the interviewed organization was between $7.7 million and $11.2 million. Over the three-year period, the total testing improvement benefit equaled a present value of $23,817,421. The interviewed organization validated a range of variables relating to testing improvements. While averages from these ranges were utilized to calculate the testing improvement benefit, it was necessary to account for the variability within those ranges; therefore, this benefit was risk-adjusted and reduced by 10%. The risk-adjusted total benefit resulting from testing improvements over the three years was $21,435,679, or about $4.1 million per statistically significant improvement test. TABLE 2 Testing Improvements Ref. Metric Calculation Initial Year 1 Year 2 Year 3 B1 Average unique visitors per test 3,500,000 3,500,000 3,500,000 B2 Average conversion rate per test 2.58% 2.58% 2.58% B3 Average order value per test $75 $75 $75 B4 B5 Average order value increase per test Average testing revenue improvements per test 12.5% 10.5% 8.5% B1*B2*B3*B4 $846,563 $711,113 $575,663 B6 Average gross margin Assumption 25% 25% 25% B7 Average margin increase per test B5*B6 $211,641 $177,778 $143,916 B8 Tests per year B9 B10 Percent of total tests with statistically significant results Percent of statistically significant tests with improvements 30% 30% 30% 25% 25% 25% B11 Average test duration B12 Annualization multiplier 365 / B11 * 50% Bt Testing improvements B7*B8*B9*B10*B12 $0 $11,265,454 $9,462,981 $7,660,509 Risk adjustment 10% Btr Testing improvements (risk-adjusted) $0 $10,138,909 $8,516,683 $6,894,458

13 13 Labor Avoidance In Search Replatform A large benefit realized by the interviewed organization was the amount of developer rework time savings realized after issues were identified. This metric was not included in the total benefits of this analysis due to the lack of information being tracked; however, an example use case of how labor was avoided in the organization s search replatform initiative is expressed below. The previous search feature was created over many years and included multiple investments to optimize through custom configurations and rules. To recreate these rules and configurations on a new platform would have taken three full-time developers an estimated 43 weeks, or a total of 5,160 development hours. With SiteSpect Origin Experiments, algorithms could be created and quickly tested to ensure similar or better performance was realized. In this example, SiteSpect Origin Experiments enabled the interviewed organization to take a new approach in migrating onto a new search tool, saving it approximately $387,000. This was one use case of labor savings described by the interviewed organization; however, others existed but were not specifically identified. TABLE 3 Labor Avoidance In Search Replatform Ref. Metric Calculation Year 1 XX1 Average weeks of configuration and rules creation saved XX2 Developers required per week 3 XX3 Total hours per week 40 XX4 Total development hours saved XX1*XX2*XX3 5,160 XX5 Average developer cost per hour $75 XXt Labor avoidance in search replatform XX4*XX5 $387, Total Benefits Table 4 shows the total of all benefits across the two areas listed above, as well as present values (PVs) discounted at 10%. Over three years, the interviewed organization expects risk-adjusted total benefits to be a PV of more than $39.9 million. TABLE 4 Total Benefits (Risk-Adjusted) Present Value Ref. Benefit Category Initial Year 1 Year 2 Year 3 Total Atr Testing risk reduction $0 $9,733,352 $7,300,014 $4,866,676 $21,900,043 $18,537,978 Btr Testing improvements $0 $10,138,909 $8,516,683 $6,894,458 $25,550,050 $21,435,679 Total benefits (riskadjusted) $0 $19,872,261 $15,816,698 $11,761,134 $47,450,093 $39,973,657

14 14 COSTS The interviewed organization experienced a number of costs associated with SiteSpect Origin Experiments: SiteSpect subscription. SiteSpect professional services. Internal testing and analysis resources. These represent the mix of internal and external costs experienced by the interviewed organization for software subscription and ongoing testing and analysis with SiteSpect Origin Experiments only. SiteSpect Subscription Software subscription costs for SiteSpect Origin Experiments were calculated based on a percentage of total testing use. Assuming 15% of all SiteSpect tests utilized the Origin Experiments capability, the monthly subscription cost was estimated at $7,250. It is important to note that the initial implementation period was immaterial to the analysis and therefore excluded. Over the three-year period, the interviewed organization incurred subscription costs for SiteSpect Origin Experiments totaling a present value of $216,356, or $87,000 per year. Subscription costs can vary from organization to organization, considering different licensing agreements, what other products may be licensed from the same vendor, and other discounts. To compensate, this cost was riskadjusted up by 5%. The risk-adjusted cost of the SiteSpect Origin Experiments subscription over the three years was $227,174. See the section on Risks for more detail TABLE 5 SiteSpect Subscription Ref. Metric Calculation Initial Year 1 Year 2 Year 3 C1 Monthly estimated SiteSpect Origin Experiments subscription cost $7,250 $7,250 $7,250 C2 Months per year Ct SiteSpect subscription C1*C2 $0 $87,000 $87,000 $87,000 Ctr Risk adjustment 5% SiteSpect Origin Experiments subscription (risk-adjusted) $0 $91,350 $91,350 $91,350 SiteSpect Professional Services SiteSpect professional services were utilized by the interviewed organization to help set up and analyze tests. On average, the professional services cost per test is $750. As 70 SiteSpect Origin Experiments tests were run each

15 15 year, the total cost per year was $52,500. Over the three-year period, the total present value for professional services equaled $130,560. Professional services needs for each test can vary based on the complexity of the test and on the sophistication of an organization s digital team. To compensate, this cost was risk-adjusted up by 5%. The risk-adjusted cost of SiteSpect professional services over the three years was $137,088. TABLE 6 SiteSpect Professional Services Ref. Metric Calculation Initial Year 1 Year 2 Year 3 D1 Tests per year D2 Average professional services cost per test $750 $750 $750 Dt SiteSpect professional services D1*D2 $0 $52,500 $52,500 $52,500 Dtr Risk adjustment 5% SiteSpect professional services (risk-adjusted) $0 $55,125 $55,125 $55,125 Internal Testing And Analysis Resources To support the 70 tests per year, a team of eight developers and analysts dedicated between 35% and 45% of their time to creating, running, and analyzing the results of SiteSpect Origin Experiments tests. These developers and analysts roles did not change when SiteSpect Origin Experiments was implemented; rather, their jobs were made simpler by utilizing the SiteSpect solution. Nonetheless, the effort involved in developing, running, and analyzing tests is calculated assuming a fully loaded annual salary of $100,000 per full-time equivalent (FTE). This equates to $360,000 in Year 1 and decreases to $280,000 in Year 3 as these developers and analysts become more efficient. Over the three-year period, the present value cost of internal testing and analysis is $802,104. The average salary and effort required by each FTE can vary from organization to organization. To account for this variability, a 15% risk-adjustment was applied to this cost. The risk-adjusted cost of internal testing and analysis over the three years was $922,419.

16 16 TABLE 7 Internal Testing And Analysis Resources Ref. Metric Calculation Initial Year 1 Year 2 Year 3 E1 Number of testing developers and analysts E2 E3 Et Etr Average annual fully loaded salary Percent of time allocated to SiteSpect Origin Experiments tests Internal testing and analysis resources Risk adjustment 15% Internal testing and analysis resources (risk-adjusted) $100,000 $100,000 $100,000 45% 40% 35% E1*E2*E3 $0 $360,000 $320,000 $280,000 $0 $414,000 $368,000 $322,000 Total Costs Table 8 shows the total of all costs as well as associated present values (PVs), discounted at 10%. Over three years, the interviewed organization expects total costs to be a PV of a little more than $1.28 million. TABLE 8 Total Costs (Risk-Adjusted) Ref. Cost Category Initial Year 1 Year 2 Year 3 Total Present Value Ctr SiteSpect subscription $0 $91,350 $91,350 $91,350 $274,050 $227,174 Dtr SiteSpect professional services $0 $55,125 $55,125 $55,125 $165,375 $137,088 Etr Internal testing and analysis $0 $414,000 $368,000 $322,000 $1,104,000 $922,419 Total costs (risk-adjusted) $0 $560,475 $514,475 $468,475 $1,543,425 $1,286,681 FLEXIBILITY Flexibility, as defined by TEI, represents an investment in additional capacity or capability that could be turned into business benefit for some future additional investment. This provides an organization with the right or the ability to engage in future initiatives but not the obligation to do so. There are multiple scenarios in which a customer might choose to implement SiteSpect Origin Experiments and later realize additional uses and business opportunities. Flexibility would also be quantified when evaluated as part of a specific project (described in more detail in Appendix A). While this was not specifically called out by the interviewed organization, a benefit that other organizations have realized from SiteSpect Origin Experiments is the ability to evaluate the performance of different vendor solutions before making a purchasing decision on digital products.

17 17 Another area of flexibility already mentioned in this study is the ability for SiteSpect Origin Experiments to quickly identify issues and, in conjunction with other software, isolate those issues much more quickly to be remediated. For example, as an update to the customer login feature was released, SiteSpect Origin Experiments quickly identified a drop in conversion for a large portion of customers. Once the code was pulled back, utilizing different software, the code causing the issues was identified and quickly fixed. After the code was pushed back into production, further testing and analysis using SiteSpect could be performed. Utilizing these software solutions together helped create a more efficient feedback process. Furthermore, the interviewed organization stated that SiteSpect Origin Experiments enables and supports Agile development and continuous release cycles by equipping teams with a capability that allows them to streamline releases, identify issues quickly, and make the necessary adjustments before moving onto the next iteration. These flexibility options were not quantified but were important to describe. RISKS Forrester defines two types of risks associated with this analysis: implementation risk and impact risk. Implementation risk is the risk that a proposed investment in the SiteSpect solution may deviate from the original or expected requirements, resulting in higher costs than anticipated. Impact risk refers to the risk that the business or technology needs of the organization may not be met by the investment in the SiteSpect solution, resulting in lower overall total benefits. The greater the uncertainty, the wider the potential range of outcomes for cost and benefit estimates. TABLE 9 Benefit And Cost Risk Adjustments Benefits Adjustment Testing risk reduction 10% Testing improvements 10% Costs Adjustment SiteSpect subscription 5% SiteSpect professional services 5% Internal testing and analysis 15% Quantitatively capturing implementation risk and impact risk by directly adjusting the financial estimates results provides more meaningful and accurate estimates and a more accurate projection of the ROI. In general, risks affect costs by raising the original estimates, and they affect benefits by reducing the original estimates. The risk-adjusted numbers should be taken as realistic expectations since they represent the expected values considering risk. The following impact risks that affect benefits are identified as part of the analysis: Testing risk reduction average order value avoidances were based on a range of tests and use cases. In many instances, the results can be different from the average utilized in the calculation. Testing improvement average order value increases were also based on a range of tests and use cases. In many instances, the results can be different from the average utilized within the calculation.

18 18 The following implementation risks that affect costs are identified as part of this analysis: SiteSpect Origin Experiments subscription costs can vary based on discounting and usage. SiteSpect professional services needs and costs can fluctuate based on a digital team s sophistication and needs. The annual salary and amount of dedicated time per FTE can vary between organizations and therefore was risk-adjusted to account for this variability. It s important to note that these were the benefits that were validated and experienced by the interviewed organization; SiteSpect shared ranges of benefits based on its customer data that coincided with these ranges. Table 9 shows the values used to adjust for risk and uncertainty in the cost and benefit estimates for the interviewed organization. Readers are urged to apply their own risk ranges based on their own degree of confidence in the cost and benefit estimates.

19 Cash flows 19 Financial Summary Table 10 below shows the risk-adjusted benefits, costs, NPV, and ROI. These values are determined by applying the riskadjustment values from Table 9 in the Risks section to the unadjusted results in each relevant cost and benefit section. FIGURE 3 Cash Flow Chart (Risk-Adjusted) $50,000,000 Financial Analysis (risk-adjusted) $40,000,000 $30,000,000 $20,000,000 $10,000,000 $0 ($10,000,000) Initial Year 1 Year 2 Year 3 Total costs Total benefits Cumulative total TABLE 10 Cash Flow (Risk-Adjusted) Summary Initial Year 1 Year 2 Year 3 Total Present Value Total costs $0 ($560,475) ($514,475) ($468,475) ($1,543,425) ($1,286,681) Total benefits $0 $19,872,261 $15,816,698 $11,761,134 $47,450,093 $39,973,657 Total $0 $19,311,786 $15,302,223 $11,292,659 $45,906,668 $38,686,976 ROI 3,007%

20 20 SiteSpect Origin Experiments: Overview The following information is provided by SiteSpect. Forrester has not validated any claims and does not endorse SiteSpect or its offerings. SiteSpect allows you to optimize the entire digital user experience, from the front end how it looks to the back end how it works. The SiteSpect Origin Experiments capability lets you test the features, functionality, algorithms, pricing, and infrastructure changes that may not be directly visible to end users. These experiments help you measure, analyze, and quantify the effect of every type of change. You can use SiteSpect Origin Experiments to execute a broad array of experiments, including: New feature releases. Soft launch new features to a subset of users, and learn how they respond in real time. The results of each test can help you refine features more quickly and deliver enhancements that resonate with your users. Search optimization. Search is one of the most visited areas of any website. Optimize your site s search algorithm. You can make head-to-head comparisons of the effectiveness of different versions of your own or other vendors search algorithms. Complete site redesigns. With a site redesign comes risk. When you change how your site visitors interact and find information, you affect the user experience. Compare the new site design with your current design to ensure the changes help you achieve your business goals. Every time you release your site, wrap a test around it to discover the impact of your redesign. Platform change. When your vendor upgrades the platform or you want to change providers, test before the change. You can test how your site performs on your current platform versus the proposed platform. Armed with information, you can make the best choice for your business. Process flow and functionality. Optimize your process flows through experimentation. The checkout process or form sign-ups are examples of process flow. Testing can tell you the best length of your forms and number of steps in the process. Architecture change. Technology advancements offer new alternatives for deploying your website. One recent trend is moving to a single-page application (SPA) to deliver a more app-like experience. An architecture change is pervasive, so be sure to test how the new architecture affects your KPIs. Network optimization. Compare different network configurations to see which options drive both performance and business impact (CDNs, data centers, etc.) SiteSpect makes Origin Experiments possible through its patented approach to optimization. SiteSpect customers using Origin Experiments have realized significant business impact.

21 21 Appendix A: Total Economic Impact Overview FRAMEWORK ASSUMPTIONS Table 11 provides the model assumptions that Forrester used in this analysis. The discount rate used in the PV and NPV calculations is 10%, and the time horizon used for the financial modeling is three years. Organizations typically use discount rates between 8% and 16% based on their current environment. Readers are urged to consult with their respective company s finance department to determine the most appropriate discount rate to use within their own organizations. TABLE 11 Model Assumptions Ref. Metric Calculation Value F1 Average gross margin 25% F2 Average developer cost per hour $75 F3 Average developer and analyst fully loaded salary $100,000 Total Economic Impact is a methodology developed by Forrester Research that enhances a company s technology decisionmaking processes and assists vendors in communicating the value proposition of their products and services to clients. The TEI methodology helps companies demonstrate, justify, and realize the tangible value of IT initiatives to both senior management and other key business stakeholders. TEI assists technology vendors in winning, serving, and retaining customers. The TEI methodology consists of four components to evaluate investment value: benefits, costs, flexibility, and risks. BENEFITS Benefits represent the value delivered to the user organization IT and/or business units by the proposed product or project. Often, product or project justification exercises focus just on IT cost and cost reduction, leaving little room to analyze the effect of the technology on the entire organization. The TEI methodology and the resulting financial model place equal weight on the measure of benefits and the measure of costs, allowing for a full examination of the effect of the technology on the entire organization. Calculation of benefit estimates involves a clear dialogue with the user organization to understand the specific value that is created. In addition, Forrester also requires that there be a clear line of accountability established between the measurement and justification of benefit estimates after the project has been completed. This ensures that benefit estimates tie back directly to the bottom line. COSTS Costs represent the investment necessary to capture the value, or benefits, of the proposed project. IT or the business units may incur costs in the form of fully burdened labor, subcontractors, or materials. Costs consider all the investments and expenses necessary to deliver the proposed value. In addition, the cost category within TEI captures any incremental costs

22 22 over the existing environment for ongoing costs associated with the solution. All costs must be tied to the benefits that are created. FLEXIBILITY Within the TEI methodology, direct benefits represent one part of the investment value. While direct benefits can typically be the primary way to justify a project, Forrester believes that organizations should be able to measure the strategic value of an investment. Flexibility represents the value that can be obtained for some future additional investment building on top of the initial investment already made. For instance, an investment in an enterprisewide upgrade of an office productivity suite can potentially increase standardization (to increase efficiency) and reduce licensing costs. However, an embedded collaboration feature may translate to greater worker productivity if activated. The collaboration can only be used with additional investment in training at some future point. However, having the ability to capture that benefit has a PV that can be estimated. The flexibility component of TEI captures that value. RISKS Risks measure the uncertainty of benefit and cost estimates contained within the investment. Uncertainty is measured in two ways: 1) the likelihood that the cost and benefit estimates will meet the original projections, and 2) the likelihood that the estimates will be measured and tracked over time. TEI risk factors are based on a probability density function known as triangular distribution to the values entered. At a minimum, three values are calculated to estimate the risk factor around each cost and benefit.

23 23 Appendix B: Forrester And The Age Of The Customer Your technology-empowered customers now know more than you do about your products and services, pricing, and reputation. Your competitors can copy or undermine the moves you take to compete. The only way to win, serve, and retain customers is to become customer-obsessed. A customer-obsessed enterprise focuses its strategy, energy, and budget on processes that enhance knowledge of and engagement with customers and prioritizes these over maintaining traditional competitive barriers. CMOs and CIOs must work together to create this companywide transformation. Forrester has a four-part blueprint for strategy in the age of the customer, including the following imperatives to help establish new competitive advantages: Transform the customer experience to gain sustainable competitive advantage. Accelerate your digital business with new technology strategies that fuel business growth. Embrace the mobile mind shift by giving customers what they want, when they want it. Turn (big) data into business insights through innovative analytics.

24 24 Appendix C: Glossary Discount rate: The interest rate used in cash flow analysis to take into account the time value of money. Companies set their own discount rate based on their business and investment environment. Forrester assumes a yearly discount rate of 10% for this analysis. Organizations typically use discount rates between 8% and 16% based on their current environment. Readers are urged to consult their respective organizations to determine the most appropriate discount rate to use in their own environment. Net present value (NPV): The present or current value of (discounted) future net cash flows given an interest rate (the discount rate). A positive project NPV normally indicates that the investment should be made, unless other projects have higher NPVs. Present value (PV): The present or current value of (discounted) cost and benefit estimates given at an interest rate (the discount rate). The PV of costs and benefits feed into the total NPV of cash flows. Payback period: The breakeven point for an investment. This is the point in time at which net benefits (benefits minus costs) equal initial investment or cost. Return on investment (ROI): A measure of a project s expected return in percentage terms. ROI is calculated by dividing net benefits (benefits minus costs) by costs. A NOTE ON CASH FLOW TABLES The following is a note on the cash flow tables used in this study (see the example table below). The initial investment column contains costs incurred at time 0 or at the beginning of Year 1. Those costs are not discounted. All other cash flows in years 1 through 3 are discounted using the discount rate (shown in the Framework Assumptions section) at the end of the year. PV calculations are calculated for each total cost and benefit estimate. NPV calculations are not calculated until the summary tables are the sum of the initial investment and the discounted cash flows in each year. Sums and present value calculations of the Total Benefits, Total Costs, and Cash Flow tables may not exactly add up, as some rounding may occur. TABLE [EXAMPLE] Example Table Ref. Metric Calculation Year 1 Year 2 Year 3