Software Performance Test Plan Template

Transcription

1 Software Performance Test Plan Template

2 XBOSoft White Paper Performance Test Plan Template 2 With mobile and web-based platforms, performance is a must have. Users want instant response or they ll go elsewhere. Before performance testing can be performed effectively, a detailed plan needs to specify how performance testing will be executed from a business and technical perspective depending on the resources available (time, and labor/expertise) and the application being deployed, <application name> at the <CUSTOMER>. This performance test plan serves as a framework or template and minimizes text length by using tables and lists for clarity and conciseness. 1. Develop and confirm overall approach 2. Determine dependencies and baseline assumptions 3. Conduct pre-performance testing preparation 4. Formulate and tune performance testing scenarios 5. Execute performance tests 6. Analyze performance test metrics, results reporting, and summary 1. Develop and Confirm Overall Approach The performance testing approach will focus on the business processes supported by the <new system> implementation. This is a living document designed as a framework to be implemented with <each application> deployed by the <CUSTOMER>. As more information is discovered, and as we reach consensus on the appropriate performance testing approach, this plan will be updated. Note that for <CUSTOMER>, performance testing will take place for each application deployed within the XXX framework and application architecture. Each application has several particular characteristics suited to <CUSTOMER> requirements which may warrant assumptions, additional requirements, and criteria. The performance testing approach will focus on a logical view of the new system implementation. Within the context of the performance testing engagement, the performance testing will:

3 XBOSoft White Paper Performance Test Plan Template Focus on mitigating the performance risks for this new implementation Make basic working assumptions on which parts of the implementation need to be performance-tested and which parts shall be excluded Reach consensus on these working assumptions as well as the criteria for acceptable performance Use a performance testing tool that can replicate the expected production volumes and provide reporting based on performance criteria Use an environment that replicates the components (as they will exist in production) that will be performancetested - noting all exceptions and deviations Measure the performance of the system during performance testing and collect the metrics stated in the requirements, followed by analysis to determine where the system fails to meet performance criteria. 2. Determine Dependencies and Baseline Assumptions This section of the performance test plan describes the dependencies such as tasks that must be completed and baseline assumptions (conditions testing believes to be true) that must be met before effective performance testing can proceed. To proceed with any performance testing, <CUSTOMER> acknowledges that the following basic requirements should be met: Planning and Preparation Finding the right tool to meet your objectives and developing realistic and thorough test scenarios can take more time than the performance execution itself. 2.1 Components to be performance tested shall be completely functional Components to be performance tested shall be housed in hardware/firmware components that are representative or scaleable to the intended production systems.

4 XBOSoft White Paper Performance Test Plan Template Data repositories shall be representative or scaleable to be similar to the intended production systems Performance objectives and measurements shall be agreed upon, including working assumptions and testing scenarios. This will depend on our understanding developed from 1. Above Performance testing tools and supporting technologies shall be installed and fully operational given the test environment provided. This will depend on our understanding developed from 1. Above since different applications have communication protocols which require interpretation and therefore possibly a different suitable tool Time schedule is known so that each of the items above can be scoped accordingly according to time available. 3. Conduct Pre-Performance Testing Preparation This section of the performance test plan describes pretesting activities that should be performed before formal performance testing begins to ensure the system is ready. It's the equivalent to smoke testing in the functional testing space. Several pre-performance testing actions could be taken to mitigate any risks during performance testing: 3.1 Create utilities to push transactions through the QA environment using projected peak loads. The QA environment is the actual environment so this step should not be needed. 3.2 Create utilities to replace business-to-business transactions that are not going to be tested or will undergo limited performance. This would remove any dependencies on B2B transactions. Since this applications deployed will be for end users, this step should not be needed unless a scenario interacts with another external system. 3.3 Create utilities to replace internal components that will not be available during performance testing. This would remove any dependencies on these components. Since all systems should be operational, this should not be needed. 3.4 Test and implement appropriate performance tools on all high-volume servers. 3.5 Test and implement appropriate performance tools on client-side workstations. 3.6 Performance test shall create appropriate loads against the system following agreedupon scenarios, measurements, and evaluation criteria that include: o User actions (workflow) o Agreed-upon loads (transactions per minute) o Agreed-upon metrics (response times) 4. Formulate and Tune Performance Testing Scenarios 4.1 In-scope business processes This section of the performance test plan states which aspects of the system are deemed to be in-scope for the purposes of performance testing including. In general, each application deployed by the <CUSTOMER> will have a list of 3-5 business process scenarios. For example, some typical scenarios may include: User registration Logon/access Users browsing content Form development Form printing Form filling This business process list shall be formed in consultation with and approved by <CUSTOMER> personnel. 4.2 Out-of-scope business processes This section of the performance testing plan delineates which aspects of the system are deemed to be out-of-scope (measured) for the purposes of testing functions of the system that shall not be considered part of the performance testing scope. These include: Functions hosted by a third party -- therefore no significant performance impact.

5 XBOSoft White Paper Performance Test Plan Template Business scenarios After determining what lies in and out of scope, then we can begin to develop the business scenarios that will be tested as well as details regarding performance criteria and expectations. Confirmation of performance testing scenarios requires significant inputs from the <CUSTOMER>: Developing business scenarios o The business scenario starts as a simple textual description of the business workflow being performance-tested. o The business scenario expands to a sequence of specific steps with well-defined data requirements. o The business scenario is complete once <CUSTOMER> determines and confirms what (if any) additional data requirements are required because of the behavior of the application/servers (i.e. caching) Expected throughput (peak) o The expected throughput stating how many users are expected to be performing this activity during peak and non-peak hours, and peak and non-peak days of the week. o The expected throughput expands to a sequence of distinguishable transactions that may (or may not) be discernable to the end user. o The expected throughput is completed after other factors and assumptions are taken into account (if any) that could impact the load (i.e. load-balancing) Acceptance performance criteria (for instance: acceptable response times under various loads) o Acceptance performance criteria are stated by the business in terms of acceptable response times under light, normal and heavy system load. System load being day-in-the-life activity. o The performance testing team then restates the acceptance criteria in terms of measurable system events. These criteria are then presented to the <CUSTOMER> for acceptance. o The acceptance criteria are completed once stated criteria are signed off by <CUSTOMER> Data requirements (scenario and implementation specific) o <CUSTOMER> specifies the critical data elements that would influence the end-user experience. o These data requirements shall be expanded to include factors that might not be visible to the end user, such as caching. o Necessary data stores are created to support performance testing Define day-in-the-life loads based on the defined scenarios in a real situation. Based on the above definitions, we then develop Day In The Life Loads. This means allocating a percent of each scenario to simulate an actual situation. The table below shows an example: Day In Life Load Set Simulated Users Scenario Percent Allocation

6 XBOSoft White Paper Performance Test Plan Template Performance Test Execution Performance testing execution usually follows a linear process of events as follows: 5.1 Define performance-testing scenarios. 5.2 Execute performance tests as standalone tests to detect issues within a particular business workflow. 5.3 Execute performance scenarios as a "package" to simulate day-in-the-life activities (as shown in above) that are measured against performance success criteria. This may include a percentage of each of the activities simultaneously. For example, not all users will login at the same time. Rather, X% may be logging in while Y% may be executing a query, etc. 5.4 Report performance testing results. 5.5 Tune the system. 5.6 Repeat testing as required. 5.7 Manual testing and automated functional tests shall be conducted during performance testing to ensure that user activities are not impacted by the current load. 5.8 System monitors shall be used to observe the performance of all servers involved in the test to ensure they meet predefined performance requirements. 5.9 Post-implementation support teams shall be represented during performance testing to observe and support the performance testing efforts. 6. Analyze Performance Test Metrics, Report Results and Summary The performance test metrics need to track against acceptance performance criteria formulated as part of the performance testing scenarios. Execute performance tests using an iterative approach to verify accuracy and validate assumptions.

7 XBOSoft White Paper Performance Test Plan Template 7 If the <CUSTOMER> has defined these as performance requirements in Section 4, then this section shall be expanded to form the basis of the analysis. The most basic performance test metrics consist of measuring response time and transaction failure rate against a given performance load -- as articulated in the performance test scenarios. These metrics are then compared to the performance requirements to determine if the system is meeting the <CUSTOMER> s usi ess eed. As a result of the perfor a e testi g, results will be published as shown below (for example) and may also be accompanied by other graphs and charts: Performance Scenario (PS)/Day In Life Load (DILL) PS1 PS2 PS3 DILL1 DILL2 DILL3 DILL4 Metric 1 Name (i.e. user response time) Pass Criteria Result Pass Result Pass Result Fail Result Pass Result Pass Result Fail Result Fail Metric 2 Metric 3 Metric 4 This test plan is intended as a general format to be used for each application being deployed for the <CUSTOMER>. The specifics depend on the system and situations being tested. As a closing note, it is important to replicate expected production loads as accurately as possible. The analysis conducted will depend on those failing results and the circumstances during which those results occur. Based o Perfor a e Test Metri s a d Results Reporti g, a a alysis will e drawn as to problem areas, bottlenecks, and possible improvements needed. 7. Resources and Timelines This test plan is intended as a general guideline and can be adapted based on the application being deployed by the <CUSTOMER>. 7.1 Resources In general, the following associated relative resources are required for each Phase based on percentage allocation as depicted below.

8 XBOSoft White Paper Performance Test Plan Template 8 Step Description Time Period Notes 1 Confirm Work with <CUSTOMER> to finalize the approach approach and determine timelines, scheduling 2 Determine dependencies and baseline assumptions 3 Preperformance testing preparation 4 Formulate performance testing scenarios 5 Performance test execution, preliminary analysis, and re-confirm 6 Analyze Performance Test Metrics, Report Results and Summary (5%) (5%) (15%-20%) (20%-25%) (30%-40%) (10%-20%) and budget Work with <CUSTOMER> to validate assumptions, objectives, requirements. Given the time budgeted for the testing and the application being tested, install and implement the appropriate test tool on server and/or client depending on <CUSTOMER> requirements-application being tested. Test the tool to ensure required metrics can be obtained. Develop performance testing scenarios and Day In The Life Loads. This includes execution and re-running scenarios, adapting scenarios, and confirming results. This will include analyzing the results and writing a final report depicting bottlenecks and characterizing when these bottlenecks present risks and reduced performance. 7.2 Timelines This test plan is designed to e i ple e ted for ea h appli atio deployed i <CUSTOMER> s appli atio portfolio including: XXX. Each application deployed will be different and have varying degrees of customization with its own implementation specifics and environment. As noted in Section 2, it is important to execute standard functional testing is assumed to be already completed, so that most performance scenarios can be executed without blockage. Otherwise performance scenarios cannot adequately or accurately be executed in order to determine the true performance of the system.

9 XBOSoft White Paper Performance Test Plan Template 9 Phase Description Period 1 Period 2 Period 3 Period 4 Total 1 Develop and confirm approach 2 Determine dependencies and assumptions 3 Conduct preperformance test preparation 4 Formulate and tune performance test scenarios 5 Execute performance tests, analyze preliminary metrics, and re-execute as needed 6 Analyze results and report Totals XBOSoft Inc. 640 Rocca Ave. South San Francisco, CA Cover page photo by: Tomo Yun