Interactive presentation. Application methodology

Transcription

1 Interactive presentation Application methodology 01/15/2015

2 Release with confidence Deliver a performance your customers expect and your team can be proud of. Tap into our 20+ years experience in QA and follow a testing methodology that gets results. Why us? We ll tell you - in 90 seconds Load Impact exceeded our expectations. We expected load tests and test reports, but Load Impact really stepped up, helping to troubleshoot issues Jim Blackwelder, CTO at ROKKAN Interactive presentation Load Impact 2 (16)

3 The Load Impact Methodology Our methodology is intuitive and easy to follow. Our goal is to empower you to become an independent and self-sufficient performance expert. Go to execution phases Briefing: Get briefed on the important things to consider in preparation for performance testing and the metrics to focus on when testing has begun. Go to application briefing Planning: Outline the time plan and test schedule, technical restrictions, deadlines and deliverables. Specify test sizes, goals and data to be included. Scripting: Create user scenario scripts that dictate how the Virtual Users (VUs) will behave during a test - what they will do and what resources to load, along with any data stores used by the script. Execution: There are three phases of test execution. Each phase will employ any of six different test types. As initial results come in, we help you decide what types of tests should be re-run and what types should be added to the program. Interactive presentation Load Impact 3 (16)

4 The Load Impact Methodology Go to execution phases Results analysis: Analyze results and drill down to find the root cause of any current and potential problems. We will give you concrete suggestions on how to improve system performance. Fix & tune: Based on your test results and analysis, you will likely need to spend some time tuning your system to improve performance. The impact of those changes will then be validated with another round of testing. Debriefing: When all deliverables have been achieved according to plan, we ll conduct a debriefing with you. New plans will be made regarding on-going testing and the level of consulting or support needed. Interactive presentation Load Impact 4 (16)

5 Test execution phases There are three phases of test execution. Each phase will employ any of six different test types. Once you ve completed all three phases, we dub you Master Tester! Phase 1: set the stage for successful performance testing Phase one is designed to lay the foundation for on-going testing. The goal is to create a test program which can be followed continuously in order to benchmark performance, uncover hidden performance problems and validate performance improvements over time. Phase one goes as follows: Once user scenarios have been scripted, a series of Smoke Tests will be run in order to validate the correctness of the scripts. Smoke tests flush out obvious errors so actual performance testing can begin. The next step is to create the test configurations. Configurations determine the number of users to test with and how that number should vary throughout the test. It also sets which user scenarios to include in each test, how the load will be allocated between the scenarios, where the load should be generated from and what network types the virtual users should emulate. Interactive presentation Load Impact 5 (16)

6 Test execution phases Phase 2: create a performance baseline and find bottlenecks The goal of phase two is to build an understanding of the current situation and iterate tests to identify hidden problems. All types of tests will be employed when required. Phase two goes as follows: Based on the goals set during the planning phase, a series of Load Tests will be run to identify bottlenecks, failures and edge conditions, as well as to establish a baseline for evaluating the impact of future system changes. Next, a series of Maximum Capacity Tests will be run to determine the maximum number of concurrent users the system can accommodate while continuing to perform as expected. Once the system has been tuned to its optimal state, the next step is to Stress Test. Stress Tests determine a breaking point and reveal what happens when the systems goes beyond the goal - when it breaks. Phase 2 con t Interactive presentation Load Impact 6 (16)

7 Test execution phases Typically a lot is learned from these initial tests and, as a result, more complex scripts and configurations are created in order to dig deeper into the initial findings. This process is repeated until your goals are reached. Depending on the type of service being tested and the complexity of the backend (e.g. systems with complex workflows, document management, logistics controls, etc.), a series of Endurance Tests will be run. Depending on the type of service being test (e.g. Banks, Data Management, etc.), a series of Robustness tests will be conducted to evaluate the functionality of the service under load (e.g. failover and disaster recovery evaluations). Interactive presentation Load Impact 7 (16)

8 Test execution phases Phase 3: towards Continuous Delivery Phase three of the test execution process is designed for proactive performance management and monitoring. The goals of phase three are: Ensure system performance has been maintained after initiated changes and upgrades Ensure that performance has not degraded due to uninitiated changes (e.g. network or service provider changes) Enable proactive, data-driven decision making regarding resource deployment Move towards continuous integration and delivery by automating tests using Load Impact's API or CI plugins for Jenkins and TeamCity. Interactive presentation Load Impact 8 (16)

9 Application briefing In 20+ years in the biz, we ve worked with some of the biggest names in software and have accumulated a great deal of knowledge. Here are some of the things we advise you to consider before starting to test. Website or application Begin by understanding the type of service being tested. For the purpose of performance testing, software without an application programming interface (API) is really no different than any other website (see our e-commerce and media methodologies for examples). Testing an application with an API is different, particularly if the API is exposed and being accessed by many other systems (e.g. mobile applications, GUIs, other APIs, etc.). Creating test scripts (test cases) When it comes to APIs, it s quite common for there to be a frontend making calls to it. Use available frontend client(s) to record the usage patterns you wish to test. But remember, you don t want to load test the frontend, you want to load test the API. Use the frontend to record real-world usage behavior and accurately capture how, when, where and by whom API endpoint calls are made. When testing an API only, you will need to code test scripts manually or use a system like Postman to record the API endpoint calls you wish to test. If your service is live, you can also analyze logs to model your test scripts on this information. Remember, quite often you will need to translate real word users and clients accessing your API into technical numbers (e.g. users or clients active versus requests) to be able to configure the test correctly. Briefing con t Interactive presentation Load Impact 9 (16)

10 Application programming interfaces (APIs) When performance testing APIs, you really have two options available: you can either hammer the API endpoint in order to determine how many requests it can take before degrading, or you can attempt to emulate real world usage patterns of the API and try to capture the real-world dependencies that impact API endpoint and call performance. Hammering an API endpoint is a good simple test case for Continuous Integration - to make sure the endpoints don t deteriorate with every build. In a perfect world, an API endpoint would be completely atomized (i.e. self-contained: authentication, encryption, authorization encapsulated within the API call), but in the real world this wouldn t be efficient because of the dependencies that often exist. For example, you don t do authorization with every API call, you do that once and then you have a session - which is the efficient way to do it (note: not doing this would lead to scalability issues). This is the reason you want to capture usage patterns and not just hammer your API endpoints. In reality, things like external CRMs, logistics, payments, credit card validations, authorizations and authentications, encryptions, security frameworks, and transaction validations services all impact API performance and therefore should be included in testing. Ideally, if you have the time and resources, you should pursue both options: pure API endpoint testing early on in development and module testing later on in operations. Doing both would give you the most comprehensive view into the performance of your application - at the code and system level. Mobile devices and networks Application consumption is ubiquitous, omni-channel and increasingly mobile. Therefore, it can be challenging to realistically simulate app consumption patterns in a test. Luckily, Load Impact specializes in realistic user simulation. Begin by monitoring your current usage/call patterns using tools like Splunk or Loggly to understand how, where, when and by whom your application is consumed. Then, model this behavior in a Load Impact test to see how performance differs across devices, networks, locations, browsers, etc. Briefing con t Interactive presentation Load Impact 10 (16)

11 <> REST vs SOAP SOAP is a protocol, REST is an architectural style, and they are both used to expose APIs, predominantly in http(s) transports. Performance or load testing either can be technically challenging, but not necessarily difficult to implement. As SOAP is document oriented and the documents are XML, it will carry a heavy payload. Keep an eye on the bandwidth during testing since it quickly adds up. XML is also very CPU-expensive to parse, so keep an eye on your CPU usage on both the servers and your load generators during testing, especially if you use functions in your performance tests for traversing and manipulating the XML that rely on parsing the document. It is very common to only need small parts, if any, of entire XML documents when load testing. Therefore, it is much more efficient to treat the XML documents as plain text and use more efficient functions such as regular expressions to extract and manipulate the XML documents in your tests. Usually REST uses JSON for data and is therefore more lightweight than SOAP. This lowers the bandwidth requirements but also the CPU requirements since it is significantly easier to both parse and generate JSON data. Functions for parsing and producing JSON will save time when preparing performance tests and setting up scripts. Also, make sure the load testing solution your choose, such as Load Impact, supports all the necessary http(s) verbs (PUT, POST, DELETE, GET, PATCH) for your test requirements. Both REST and SOAP based infrastructure are often implemented on the latest frameworks, which can sometimes be very picky on request details. Therefore, it will be necessary to allow testers detailed and complete control of requests to actually implement working performance and load tests. Briefing con t Interactive presentation Load Impact 11 (16)

12 SOA architecture Service Oriented Architectures can be implemented on multitude different platforms using different protocols to communicate their services. There are a number of different transport and document protocols supporting SOA, but the most common are web services (which in reality are SOAP), and recently also REST. The individual tests are implemented as REST and SOAP tests respectively, but a SOA environment can quickly become extensive with sometimes up to several hundreds or even thousands of endpoints or services. Two strategies can be implemented to manage this from a performance testing perspective. They are not mutually-exclusive, and can in fact complement each other. The first is the end-to-end perspective, where performance tests are executed as normal scripted performance or load tests by recording and/or scripting user behavior from common use cases and driving these with test data if necessary. It will simulate user load and will exercise endpoints and services as necessary. I m important to remember, that it can be hard to analyze and find root causes for performance issues in a complex SOA system. It can also be very hard to have a complete system or even subsystem of a SOA architecture available for testing. The second strategy is to test endpoints. This is preferably done in a continuous integration manner to track improved or degrading performance by any new deployments or releases. This allows for load and performance testing atomized sub-systems in a SOA architectural system. Both strategies complement each other, but full end-to-end testing can be hard to accomplish in a test setup. Briefing con t Interactive presentation Load Impact 12 (16)

13 Integrations Some applications rely on a number of integrations, such as payments, social sharing and commenting. It s important to understand what integrations to include in your tests and why. Those that you suspect will impact performance should always be included; those that probably don t and for which you have little control over should not be included. Scalability Load testing will validate your assumptions regarding if and how your systems scale. It s important to understand that infrastructure limits are not linear - simply adding more won t necessarily multiply capacity. In our experience, even the most well designed architecture has a scalability limit and will need to be redesigned at some point. Interactive presentation Load Impact 13 (16)

14 Measure the things that matter Requests per second and response time per request For pure API testing, do not only focus on how many requests per second (req/s) the endpoint can handle, also focus on response times for the individual requests. Req/s alone will not indicate whether or not you are meeting performance goals, it s the combination of the two metrics that will. Interdependencies between multiple APIs There are interdependencies between the performance of different APIs. For example, computationally heavy APIs that are rarely called can have a major impact on the performance of simpler APIs that are called more frequently. Often you do not know the impact of the computationally heavy query on other API response times, until you test it (note: the APIS don t need to be dependent on each other, its just that they share the same infrastructure). Look at the response times for ALL the different APIs in the test, not just how many requests per second it can handle. Key transaction response times Certain transactions are critical from a business perspective, but aren t always obvious to those working with building and testing individual API endpoints. For example, being able to get a quote before submitting an order, are not mutually exclusive API calls from a business standpoint but may be so from a technical standpoint. Work with business stakeholders to define which API calls should be included in the transactions tested. Very high volume load Focus attention on response times for actual content, not stylesheets and static data as they are typically handled by a CDN or some type of caching frontend. Purposely exclude certain metrics from the test, otherwise you will drown in data from the test. Server metrics Monitor the servers and databases of the application being tested. Doing so makes your analysis faster and helps you identify performance bottlenecks; and it keeps you from wasting time investigating and fixing things that have no real impact on performance. Interactive presentation Load Impact 14 (16)

15 The six types of tests 1. Smoke Tests: These tests are run in order to uncover obvious flaws in test scripts, platforms, code, etc. before committing the resources of a full test. 2. Load Tests: These tests are run to evaluate whether or not performance goals are met and to confirm that all system issues have been identified and resolved. 3. Max Capacity Tests: These tests are run to define the maximum number of concurrent users the system can accommodate while continuing to perform as expected. 4. Stress Tests: These tests are run to determine a breaking point and to evaluate what happens when the system breaks. 5. Endurance tests: These tests are designed to find problems that occur when a system is put under slight pressure for a long period of time (e.g. memory leaks). If a business needs to be functional 24/7, Endurance Tests are essential. 6. Robustness Tests: This functional testing under load is designed to validate that a system continues to function as expected when put under abnormal pressure. They are useful for evaluating failover and disaster recover procedures. Interactive presentation Load Impact 15 (16)

16 See how others have done it Read how other software application companies have used Load Impact to analyze and improve their performance. Ushahidi is a non-profit tech company that specializes in developing free and open source software for information collection, visualization and interactive mapping. Ushahidi used Load Impact to greatly improve the performance of its software. Through comparing before and after test results it was possible to see the impact of performance optimizations done. Learn more Join these and other great software companies Get started Interactive presentation Load Impact 16 (16)