DevOps Guide: How to Use APM to Enhance Performance Testing

DevOps Guide: How to Use APM to Enhance Performance Testing

CHAPTER 1: Introduction This short ebook discusses how combining performance test automation with application performance management (APM) solutions can be a powerful enabler to implementing DevOps within the enterprise. Implementing DevOps is rapidly becoming the norm for businesses looking to improve the quality and speed of release in response to demands of the Digital Revolution. Increasing the speed and quality of release delivery is key to remaining competitive and agile to deliver on-demand business change. When partnered with performance testing, APM is well-placed to help ensure core applications continue to provide a great customer experience, even under conditions of peak load. But first, let s look at performance testing: what it is and why it s needed more than ever. DevOps Guide: How to Use APM to Enhance Performance Testing 2

Performance Testing OVERVIEW Performance testing is a practice designed to simulate and measure how stable and responsive a software application is under a particular workload. It can also serve to investigate, measure, validate, or verify other quality attributes of the system, such as scalability, reliability, and resource usage. There are different performance tests, each with their own purpose: Load Test (or Workload Test) Designed to simulate the current peak day in production Stress Test Understanding how a system reacts to extreme load conditions like 200% of current production peak, for example Capacity Test Sees if the footprint and number of servers are right-sized Device, FE, BE, end-to-end, and front-end optimization, focused on: Collapsing calls Reducing latency Minimizing bandwidth and size of resources (css, JS, images) Endurance or Soak Test Runs for an extended time to see if anything fails DevOps Guide: How to Use APM to Enhance Performance Testing 3

Performance testing is enabled by automation and is impractical as a manual testing exercise. The key capability that automation provides is the ability to record and script end-user activity, then iteratively replay this activity at scale against a target environment, all while measuring application and system performance. There are many performance test automation solutions available in today s IT marketplace (licensed or open-source) to suit budget and requirement. Many of these integrate directly with APM solutions like AppDynamics. Performance testing is its own discipline and is distinct from other forms of software testing. It is commonly referred to as non-functional, as the focus is on testing application characteristics other than pure function (i.e., scalability and availability under load). Effective performance testing relies on capturing essential non-functional requirements (NFRs), including: Business Identifying scope (strategic, tactical) Confirming expected deliverables (ensuring no surprises) Confirming client platforms to test (desktop, mobile, service consumers, batch) Identifying key use cases, test data requirements, and test scenarios for each client platform Identifying the key performance indicators (KPIs) that must be captured to accurately measure application performance Building an accurate workload model (essential) Deciding whether performance test resourcing will be sourced in-house or outsourced to a third party Technical Management and provision of suitable performance test environments Performance automation tooling selection appropriate for technical fit, skill set, and budget (ideally via POC) Monitoring approach (ideally based on APM) Data collection, analysis, correlation, and reporting Once captured, these NFRs enable the creation of test assets that are a true representation of application behavior, including: Scripted use cases that represent realistic end-user activity on browsers or mobile devices. (Alternatively, they could represent API calls from service consumers.) An accurate workload model that defines: The types of performance tests required The distribution of scripted use cases amongst performance tests and the scale of load to be generated A test environment that is appropriate for the level of testing required. For end-to-end testing, this should be a copy or close approximation of production. For other testing requirements, such as component performance testing in dev, this could be an environment of sufficient scale to test service-level agreement (SLA) compliance for a discrete subset of application functionality. DevOps Guide: How to Use APM to Enhance Performance Testing 4

MONITORING AND ANALYSIS CURRENT PERFORMANCE TESTING TRENDS Of course, simulating end-user traffic is only half the picture. We need to be able to capture and understand application performance under load, as well. Performance automation solutions have traditionally provided metrics such as: Response time of complete and nominated subsets of scripted use cases. For example, a user may log in, carry out a search, then add an item to their basket. Each of these user actions can typically be timed. The number of concurrent users at any point during a test execution. This metric is typically correlated with response time to demonstrate how increasing load impacts application response time. The number and type of errors that occur during a test run. The appearance of errors at a certain level of load is an important tell-tale sign of problems with application scalability. While these metrics are great at identifying when a problem occurs, they don t necessarily pinpoint the root cause. To identify root cause, we need to capture performance data from the full stack, including the network, hosting servers, and databases. Performance automation solutions have long provided integration with monitoring point solutions like Windows Performance Monitor or SNMP. And while this does give additional insight into application performance, it is not full-stack and only reports on the metrics selected for the servers or software components being monitored. In response to the Digital Revolution and business requirements, performance test automation solutions have evolved over time to accommodate: Increased release velocity, a catalyst for change Smaller, more frequent releases Faster time to market for new features and fixes Reduction in pain of giant monolithic deployments Customer demands: faster, better, less patience Tooling vendors now typically offer: Focused support for modern web development technologies, such as Node.js, AngularJS, SPF, and WebSockets Integration with established functional test solutions, such as Selenium Integration with test automation servers, such as Jenkins as part of CI and CD Integration with APM The last evolution is key, as integration with APM solutions like AppDynamics provides unique insight and understanding as to why an application fails to scale or respond under load and more importantly, how this impacts the end user. DevOps Guide: How to Use APM to Enhance Performance Testing 5

CHAPTER 2: Performance Challenges Undetected performance defects at any stage of the systems development life cycle (SDLC) can impact an application s ability to deliver the level of service and availability that the business and the customer expect. Shift left is a popular term to describe a change to the traditional software testing approach. In response to the widespread adoption of agile and DevOps, shift left promotes testing early and often, not just in QA but also in dev. This provides some important advantages over a traditional QA-only approach to testing: The earlier you detect defects in the SDLC, the easier and cheaper it is to fix them. By finding and fixing performance defects earlier, you reduce the amount of testing required, increasing the speed and quality of delivery. With that in mind, the next question becomes, How do you stop performance defects from creeping into the SDLC in the first place? Let s look at some common reasons why performance defects occur in the following IT stages: Design Development Testing Deployment Scalability DevOps Guide: How to Use APM to Enhance Performance Testing 6

DESIGN TESTING For design, this responsibility clearly lies with business and IT stakeholders, as well as anyone else who has input into application design. Put simply, if the application isn t designed from the ground up to scale, then it will be very unlikely to satisfy business or end-user requirements for performance. What s more, these problems may not be detected until late in the SDLC, when it typically can only be corrected with time-consuming and costly remedial action. DEVELOPMENT The traditional model for development has teams of coders assigned to groups of work items and tasks. Areas of performance risk include: Performance test automation has evolved from basic replay engines to sophisticated test and monitoring frameworks. There s certainly a lot of choice in the marketplace, including established open-source offerings like JMeter. The good news is that most businesses regardless of size now use some form of performance automation tooling within QA. The bad news is that performance testing is often implemented as a purely tactical function, lacking formal process and management when it should be an integral part of strategic IT planning. This can result in incomplete performance NFR capture and release decisions, based on incomplete or misleading test results. Lack of or minimal control over coding standards Developers not having the bigger application picture Developers not typically taking performance SLAs into account when coding Developers not conducting performance testing Developers not talking to Ops In reality, much of the above is changing, due in no small part to the widespread adoption of an agile development style. However, the picture is still patchy within many organizations on the journey from waterfall to agile. Performance considerations are often only considered at a high level even within agile-sprint planning and are frequently deferred in response to pressure to deploy. DEPLOYMENT When it comes to deployment decision-making, there is potential for considerable performance risk. If those responsible for production deployment don t have visibility into appropriate performance metrics for the release candidate under consideration, then a critical piece is missing from the data that informs their decision. In other words, there should be a clear indication of whether the application has met any performance SLAs related to response time, scalability, and availability at anticipated peak load. DevOps Guide: How to Use APM to Enhance Performance Testing 7

SCALABILITY An application that doesn t scale is highly vulnerable to unexpected peaks in load, which can result in partial or complete loss of availability. Even if it stays available, end-user performance may degrade to such an extent that the application may as well be unavailable. The emergence of the cloud has enabled many applications that were not originally designed with scalability in mind to continue to provide a service to end users under increasing load. This temporary fix may provide some breathing space, but it inevitably incurs increasing costs, as expanding numbers of cloud instances need to be provisioned to support demand. Lack of scalability is more often than not a result of a design miss, so it is unlikely to be trivial (or inexpensive) to correct. Fig 1 Application Performance Challenges System Under Test EUE Mobile Users Capacity Web Server Tier Network Issues EUE Web Users Code Quality App Server Tier Contention Config Issues Database Cluster EUE LAN Users (No) SQL Performance DevOps Guide: How to Use APM to Enhance Performance Testing 8

CHAPTER 3: The Process Now that we ve covered the performance risks and how they re present across the SDLC, let s explore how we can address these risks using performance testing best practices and application performance management (APM). DevOps Guide: How to Use APM to Enhance Performance Testing 9

Application Performance Management (APM) A LITTLE HISTORY APM has evolved from simple code profilers commonly used in development to full-stack analysis of software applications across the SDLC from last-mile to first-mile with increasing focus on understanding end-user experience. The transactional view of user actions (from browser click to database call) provided by APM is the most effective way to triage and resolve application performance problems as part of performance testing or production monitoring. APM is rapidly becoming the solution of choice for centralized performance and availability management, as well as the monitoring of core applications and end users. More importantly, this cross-sdlc capability makes APM a natural choice for a partnership with performance test automation. Let s revisit the IT stages we discussed in Chapter 2 to see how APM and performance testing can help mitigate performance risk and enable DevOps. DevOps Guide: How to Use APM to Enhance Performance Testing 10

ADDRESSING RISK WITH PERFORMANCE TESTING Design At the design stage, there generally isn t anything tangible to work with. However, we can still leverage performance testing and APM to inform our design choices by running trial Proof of Concepts (POCs) on technology candidates for key application components. Performance defects can be identified early and corrected before being promoted to higher levels of deployment and testing. In addition, performance testing can be made part of the overnight batch job, and APM can be used to compare performance between code releases and provide trigger pass/fail alerts as part of test automation server integration. One example includes comparing NoSQL database performance. APM solutions like AppDynamics provide support for a wide range of NoSQL databases, including comprehensive monitoring and analysis. From a DevOps perspective, a strategy can also be put in place for APM monitoring across the SDLC to identify SLA KPIs that include: Application KPIs Service KPIs Infrastructure KPIs These can then be rolled forward and curated as required when applications progress through the SDLC. Develop During development, an APM solution like AppDynamics can add value to DevOps simply by implementing an early performance-testing capability at the software-component level, then deploying APM into dev environments. In dev, we usually don t have the luxury of something that can be tested end-to-end, but we can still do useful performance testing. Our focus shifts to code quality, contention issues, and API performance all of which can be easily monitored by performance testing and APM. Te st QA tends to be the first opportunity to carry out end-to-end testing, so the full-stack view of application performance provided by APM can be leveraged to great effect, including the all-important end-user experience. When performance testing in dev becomes a normal process, iterative QA performance testing can be significantly reduced and become more of a confirmation that performance metrics remained unaffected by end-toend deployment. Deploy As discussed, a common failing when deciding on the suitability of a release candidate for deployment is the lack of performance metrics to help inform the decision process. When performance testing is implemented as part of the formal process, performance metrics are an output of every stage of testing across the SDLC. A formal report demonstrating application KPI compliance should be a mandatory input to the release decision-making process. APM s additional performance insight provides greater confidence that an application is ready for deployment. In addition, the KPIs rolled forward from design and further curated by testing in dev and QA can be used as the basis for production monitoring ideally provided by APM. DevOps Guide: How to Use APM to Enhance Performance Testing 11

Scale Finally, the ability for an application to cope with peaks in load (planned and unplanned) is vitally important. Performance testing on its own can demonstrate the capacity limits of an application but won t necessarily tell you why it fails to scale. By combining performance testing with APM, application scalability can be directly mapped to host database and code-level behavior, helping to pinpoint the reasons for not supporting the level of load required. This enhanced view can also be used to explore the upper limits of application scalability to reliably inform planning for any future increase in infrastructure capacity. Fig 2 Performance Testing and APM Across the SDLC Monitor Design Performance Testing and APM Develop Scale Test Deploy DevOps Guide: How to Use APM to Enhance Performance Testing 12

CHAPTER 4: Measuring Success Finally, let s examine how we measure success. We ve discussed performance challenges and how to mitigate risk and enable DevOps using performance testing and APM, but how can we measure the effectiveness of our approach? DevOps Guide: How to Use APM to Enhance Performance Testing 13

Checklist to Enable Success In order to measure success, we first need to confirm we re following best practices with performance testing and APM. Let s return to our IT stages for a final time and checklist the key actions. DESIGN (FOR PERFORMANCE) Make performance considerations a fundamental requirement of application design Define a set of performance KPIs as appropriate for each new application or set of functionality: Response time Concurrency Throughput Availability Assess all change requests for performance impact DEVELOP (FOR PERFORMANCE) Performance test in dev (i.e., shift left) Make performance testing integrated with APM a part of standard process Make performance testing integrated with APM part of the overnight batch automation testing Track the number of performance defects as distinct from functional defects Monitor if they are increasing or decreasing Make performance considerations an integral part of sprint planning Use APM to monitor dev test environments TEST (FOR PERFORMANCE) Performance test in QA Make performance testing integrated with APM standard process Track the number of performance defects as distinct from functional defects Monitor if they are increasing or decreasing Use APM to monitor QA test environments DEPLOY (FOR PERFORMANCE) Make formal performance metrics based on design KPIs a mandatory input to the release decision process Use APM to monitor core applications in production Use APM to monitor the end-user experience for all core applications Use design KPIs as a baseline measure of application performance in production Measure the quality of each release in terms of reported performance defects SCALE (FOR PERFORMANCE) Know what the scalability requirements for applications are at release and in relation to projections for growth Design applications to scale Reviewing the above on a regular basis should provide a useful guide to implementing effective application performance management within your enterprise. DevOps Guide: How to Use APM to Enhance Performance Testing 14

The ultimate measure of your success will be determined by tracking three KPIs: Fig 3 Performance Defect Discovery Shift Left 1. The number and location of performance defects: Ideally you should see the weighting of defect discovery reducing in number and moving away from production and QA towards design and dev as increasing DevOps-driven performance awareness takes root and matures. 10 9 Ideal Performance Defect Discovery Trend Shift Left 2. The speed and quality of release: Increased release velocity and quality should confirm the benefit of shift left discovery and reduction in performance defect numbers. 3. Happy end users: Software releases that consistently deploy on time and meet or exceed business and end user expectations on performance are final confirmation that application performance management in your enterprise is in a good place. 8 7 6 5 4 3 2 1 0 Design DEV QA Production Reducing DevOps Guide: How to Use APM to Enhance Performance Testing 15

CONCLUSION The value of performance testing and the ability to ensure application performance is vital and this testing can be greatly enhanced with APM. We ve explored how APM-provided enhanced performance insight can be leveraged across the SDLC to improve the speed and quality of delivery. We ve also looked at common performance challenges, and the reasons they occur, and we ve offered suggestions as to how they may be avoided. APM has certainly come of age and is now an essential component to building a performance testing plan. Learn more about how AppDynamics can improve performance testing for your business. DevOps Guide: How to Use APM to Enhance Performance Testing 16