Getting started with load & performance testing

Transcription

1 Getting started with load & performance testing How to incorporate into my company? Where to keep an eye on? Essen,

2 Speaker Dr. Jan Sickmann profi.com AG IT Consultant Mobile:

3 Agenda 1. Why Load & perfomrance testing? 2. Stairway to load & performance testing 3. Choose the right tools 4. Execute and analyze the results 5. Summary

4 Agenda 1. Why Load & perfomrance testing? 2. Stairway to load & performance testing 3. Choose the right tools 4. Execute and analyze the results 5. Summary

5 Why load & performance testing? Prevent costly failure of mission-critical applications Assure performance and functionality under real world conditions Locate potential performance issues before your customers do Avoid wrong design and infrastructure decisions Reduce development time Reduce infrastructure costs

6 Types of performance testing Component testing Load testing Stress testing Volume testing Find the behavior and performance of each tier Determine if the system handles anticipated real-world load Find system break point; measure if system environment is properly configured for unexpected, high-transaction volume Check system stability under extended periods of load

7 Examples of Performance Test Objectives Application response time How long does it take to complete a task? Coniguration sizing Which configuration provides the best perfomance level? Regression Does a new version of the software adversely affect response time? Reliability How stable is the system under heavy work load? Capacity planning At which point does performance degradation occur? Bottleneck identification What is the cause of the degrading performance?

8 Functional vs. non functional testing Goal Functionality Functional testing Example Do business processes still match the requirements after an implementation? Goal Performance under load Stability Functionality under load Non functional testing Example Does the response time at 1,000 parallel requests match the requirements? Does 1,000 parallel requests influence the system over a certain time? Do all business processes still work correctly under load?

9 Functional vs. non functional testing Goal Functionality Functional testing Example Do business processes still match the requirements after an implementation? Keep an eye on basic performance (single user) even during functional testing! Goal Performance under load Stability Functionality under load Non functional testing Goal Does the response time at 1,000 parallel requests match the requirements? Does 1,000 parallel requests influence the system over a certain time? Do all business processes still work correctly under load?

10 Functional vs. non functional testing Goal Functionality Functional testing Example Do business processes still match the requirements after an implementation? Keep an eye on basic performance (single user) even during functional testing! Goal Performance under load Stability Functionality under load Non functional testing Goal Does the response time at 1,000 parallel requests match the requirements? Does 1,000 parallel requests influence the system over a certain time? Do all business processes still work correctly under load?

11 Agenda 1. Why Load & perfomrance testing? 2. Stairway to load & performance testing 3. Choose the right tools 4. Execute and analyze the results 5. Summary

12 Get the right people together Project manager Developers Testmanager Test engineer IT operations department Competent department

13 Get the right people together Coordinating planning, test exection and analysis Project manager Coordinating the whole development project. Creating test scripts and executes tests Name load critical functions. Receive test results. Fixing application Infrastructure information. What to monitor? Support monitoring Developers Testmanager Test engineer IT operations department Competent department User stories, use cases, numbers for general usage and concurrent users

14 Get the right people together Coordinating planning, test exection and analysis Developers IT operations department Project manager Testmanager Coordinating the whole development project. Creating test scripts and executes tests Get in touch before load testing for deriving test acceptance criteria and load models Name load critical functions. Receive test results. Fixing application Infrastructure information. What to monitor? Support monitoring Test engineer Competent department User stories, use cases, numbers for general usage and concurrent users

15 Defining Goals Start with conceptual goals Jotting down goals that can t be measured will allow later filtering to create more focused goals Examples: Responsivness of a search function: Are we able to get search results within a reasonable time? System administrator s concern if the Update transaction functions during heaving usage. etc

16 Quantifying Goals Conceptual goal The search must be fast enough Confirm that save function works under heavy traffic Quantitive goal Search transaction time 5 sec for 2,000 concurrent users during peak hours (8-10 a.m.) Achive 300 concurrent users for save function at peak time (noon)

17 Gathering system usage information Why? Allows to Decide which business processes to test Isolate peak loads and peak load times Document user actions and valid input data for each business process How! Using the site, first-hand Consulting with administrators (IT operations department) Consulting with executives (competent department) Researching competitor s sites

18 Analyzing the system under test Why? Allows you to setup monitors Allows you to effectively coordinate with backend experts Provides system information that helps isolate performance problems How! Ask System Administrators Backend experts Application experts Database Administrators

19 Identifying business processes to test Mission-Critical Business Process Business Processes that are crucial in day-to-day operations Heavy Throughput Heavy throughput business processes may not be mission-critical but are very popular Dynamic Content Dynamic content consists of sever requests that are customized for each user

20 Identifying business transactions to test What are critical transactions (key transactions)? Transactions executed very often (heavy throughput) Transactions excetuted less often but creating a high system load Transactions executed less often but the response time should be measured anyway Transaction that are mission critical (in case of costs, public reputation, etc.) Finally, create (artifical) business processes from these transactions

21 Identifying business transactions to test Mission critical transactions: E-commerce (add to cart, orders, payments,...) E-banking (transfers, trading,...) Get info from: Application experts Competent department Management teams

22 Identifying business transactions to test Heavy throughput transactions: Homepage viewing Logins Site navigation Get info from: Competent department Server statistics

23 Identifying business transactions to test Load intensive transactions: Searches Dynamic page generation Streaming media Get info from: Application experts Administartors IT operations department

24 Business process matrix Business process Users (typical day) Users (peak time) Dynamic content Mission critical Test Sign in 70/hr 210/hr Low High? Creating new accounts Searches for artists View artist s page Purchase MP3s 10/hr 15/hr Moderate Low? 130/hr 180/hr Moderate Moderate? 20/hr 30/hr Moderate High 40/hr 90/hr High High?

25 Defining a real world load Concurrency = Number of users acting (!) on the application at the same time (!) Application level How many users are active on the system? Business process level How many users are buying MP3s? Transaction level How many users press Buy MP3 now?

26 Defining a real world load Do not focus on business processes/transactions or concurrent users alone! Better: How many users execute which transactions (business process) during which time? Dervive from business process matrix and statistics Create an extended transaction/business process matrix

27 Time recordings saved Defining a real world load Example: Save time recordings Hours

28 Creating a transaction matrix

29 Creating a transaction matrix

30 Documenting user steps and input data

31 Determining valid test data Sources: Application Data Data is resident in the application s database Examples: ID numbers and passwords User-Generated Data Originates with the user Examples: new unique ID or address External Data Data is unknown before the application is run Examples: confirmation and purchase order numbers

32 Analyzing the system Understanding system components Resources Web Server, Application Server, Database, Middleware, SAN, Load Balancer, Firewall Type Apache, IIS, WebLogic, Oracle, MS SQL, SAP PO, F5, PaloAlto Operating System Linux, Unix, Windows Server 2008/2012, Windows 7/10, Embedded

33 Analyzing the system Mapping business processes to components Example: Searches -> Web Server, Application Server, Database, SAN Homepage -> Web Server Purchase orders -> Web Server, Application Server, Database, SAN

34 Analyzing the system Mapping business processes to components Example: Searches -> Web Server, Application Server, Database, SAN Homepage -> Web Server Purchase orders -> Web Server, Application Server, Database, SAN Draw appropriate monitoring conclusions

35 Monitoring system components Firewall not configured properly Load balancer not distributing properly Faulty Web server from cluster Database is slow or unable to feed data Application server cannot handle the request

36 System performance monitors System resource monitors Network monitors Firewall monitors Webserver monitors Application server monitors Database monitors ERP/CRM monitors

37 Test environment Mirrors the production environment Is dedicated to performance testing only (no interference with production users) Allows data to be written, read, destroyed Allows test system to be rebooted Allows to run the business processes correctly (application code freeze) Must contain sufficient hardware to generate the defined load

38 Agenda 1. Why Load & perfomrance testing? 2. Stairway to load & performance testing 3. Choose the right tools 4. Execute and analyze the results 5. Summary

39 LoadRunner on-premise Controller On-premise Load Generators Appplication under test

40 LoadRunner on-premise + cloud LGs Cloud Load Generators Controller Appplication under test

41 LoadRunner on-premise + cloud LGs Cloud Load Generators Controller On-premise Load Generators Appplication under test

42 StormRunner + on-premise LGs StormRunner Cloud Load Generators On-premise Load Generators Appplication under test

43 StormRunner StormRunner Cloud Load Generators Appplication under test

44 Which solution should I choose? LoadRunner on premise LoadRunner + cloud LGs StormRunner More frequent testing Short term engagements Quick, high volume tests Determing baseline performance Remote Load generation Go-to platform for troubleshooting bottlenecks Web only Detailed reporting and analysis Limited reporting

45 Which solution should I choose? HPE Loadrunner with ALM Integration LOADRUNNER ENVIRONMENT Load Generator APPLICATION QC / ALM Controller Virtual Users Web Server Application Server Database Server HTTP, SOAP, ORACLE, CITRIX ICA, FTP, SMTP, MAPI, LDAP, JAVA, Peoplesoft, SAP LASTTEST METRICS: CPU RAM IO MONITORING METRICS: CPU RAM DISK IO PROCESS

46 Which solution should I choose? Integration HPE Performance Center PERFORMANCE CENTER ENVIRONMENT QC / ALM Performance Center Controller Load Generator Virtual Users APPLICATION Web Server Application Server Database Server HTTP, SOAP, ORACLE, CITRIX ICA, FTP, SMTP, MAPI, LDAP, JAVA, Peoplesoft, SAP LASTTEST METRICS: CPU RAM IO MONITORING METRICS: CPU RAM DISK IO PROCESS

47 Which solution should I choose? Performance Center Very frequent (continous) testing All features like LoadRunner Easy test management within ALM Even more scalable Parallel and scheduled tests

48 Agenda 1. Why Load & perfomrance testing? 2. Stairway to load & performance testing 3. Choose the right tools 4. Execute and analyze the results 5. Summary

49 Phases of load and performance testing Planning Preparation Exection Analysis Defining goals Identify responsibles Identify testing capabilites of the application (PoC) Set-up test environment, Identify project time frame Defining monitors Configuring test environment Access test environment Creating test scripts Define load and iterations Creating test data Configure LoadRunner scenario Executing load tests Inform customer if test fails Documentation of findings and issues Documentation of results Validate test runs Central repository for documentation and results Report and recommendations Results presentation

50 Test case density 1/s Phases of ideal load test Scaling Stability Overload Scaling Test case density ~ load increases with number of users untill saturation of bottleneck device User

51 CPU load / % Phases of ideal load test 100 Scaling Stability Overload Scaling Test case density ~ load increases with number of users untill saturation of bottleneck device User

52 Test case density 1/s Phases of ideal load test 250 Scaling Stability Overload Stability Test case density ~ load = const User

53 Response time / s Phases of ideal load test 20 Scaling Stability Overload Stability Response time increases slightly User

54 Response time / s Phases of ideal load test 20 Scaling Stability Overload Overload Response time acts impredictable User

55 Analyzing results Is there enough test hardware available in the test environment? Benchmark business processes against the testing hardware Take a business process and execute a small number of users against the application. Validates the functionality of the business process Potentially exposes unknown data dependencies Evaluate the testing infrastructure against the footprint. Do I have enough hardware to generate the user load? Do I have enough memory? Do I have enough CPUs?

56 CPU Utilization [%] Transaction response time [s] Analyzing results Example: Response time and CPU consumption RT=f(Users) Step1 No. of VUsers Transaction Response Time CPU Utilization No. of VUsers CPU Utilization [%] 80 % 4 s 1 s Step 2 No. of VUsers 0 10 % time 100 % CPU=f(Users) Testcase 50 % 10 % No. of VUsers

57 Analyzing results Don t forget to document the test execution and results Get togehter with application experts and administrators to analyze the findings and draw conclusions Derive recommendations for performance optimization Derive sizing recommendations Discuss the lessons-learned

58 Agenda 1. Why Load & perfomrance testing? 2. Stairway to load & performance testing 3. Choose the right tools 4. Execute and analyze the results 5. Summary

59 Summary Start early Consider performance as part of your application development Get the right people involved Define your goals (even non performance goals but project goals) Analyse your system to test, define load metrics and choose tooling Set-up monitors Execute the load tests Analyse the results and fix your application

60 Questions?