Getting started with load & performance testing How to incorporate into my company? Where to keep an eye on? Essen, 2017-05-04
Speaker Dr. Jan Sickmann profi.com AG IT Consultant Email: jsickmann@proficom.de Mobile: +49 151 29806897
Agenda 1. Why Load & perfomrance testing? 2. Stairway to load & performance testing 3. Choose the right tools 4. Execute and analyze the results 5. Summary 12.06.2017 www.proficom.de 3
Agenda 1. Why Load & perfomrance testing? 2. Stairway to load & performance testing 3. Choose the right tools 4. Execute and analyze the results 5. Summary 12.06.2017 www.proficom.de 4
Why load & performance testing? Prevent costly failure of mission-critical applications Assure performance and functionality under real world conditions Locate potential performance issues before your customers do Avoid wrong design and infrastructure decisions Reduce development time Reduce infrastructure costs 12.06.2017 www.proficom.de 5
Types of performance testing Component testing Load testing Stress testing Volume testing Find the behavior and performance of each tier Determine if the system handles anticipated real-world load Find system break point; measure if system environment is properly configured for unexpected, high-transaction volume Check system stability under extended periods of load 12.06.2017 www.proficom.de 6
Examples of Performance Test Objectives Application response time How long does it take to complete a task? Coniguration sizing Which configuration provides the best perfomance level? Regression Does a new version of the software adversely affect response time? Reliability How stable is the system under heavy work load? Capacity planning At which point does performance degradation occur? Bottleneck identification What is the cause of the degrading performance? 12.06.2017 www.proficom.de 7
Functional vs. non functional testing Goal Functionality Functional testing Example Do business processes still match the requirements after an implementation? Goal Performance under load Stability Functionality under load Non functional testing Example Does the response time at 1,000 parallel requests match the requirements? Does 1,000 parallel requests influence the system over a certain time? Do all business processes still work correctly under load? 12.06.2017 www.proficom.de 8
Functional vs. non functional testing Goal Functionality Functional testing Example Do business processes still match the requirements after an implementation? Keep an eye on basic performance (single user) even during functional testing! Goal Performance under load Stability Functionality under load Non functional testing Goal Does the response time at 1,000 parallel requests match the requirements? Does 1,000 parallel requests influence the system over a certain time? Do all business processes still work correctly under load? 12.06.2017 www.proficom.de 9
Functional vs. non functional testing Goal Functionality Functional testing Example Do business processes still match the requirements after an implementation? Keep an eye on basic performance (single user) even during functional testing! Goal Performance under load Stability Functionality under load Non functional testing Goal Does the response time at 1,000 parallel requests match the requirements? Does 1,000 parallel requests influence the system over a certain time? Do all business processes still work correctly under load? 12.06.2017 www.proficom.de 10
Agenda 1. Why Load & perfomrance testing? 2. Stairway to load & performance testing 3. Choose the right tools 4. Execute and analyze the results 5. Summary 12.06.2017 www.proficom.de 11
Get the right people together Project manager Developers Testmanager Test engineer IT operations department Competent department 12.06.2017 www.proficom.de 12
Get the right people together Coordinating planning, test exection and analysis Project manager Coordinating the whole development project. Creating test scripts and executes tests Name load critical functions. Receive test results. Fixing application Infrastructure information. What to monitor? Support monitoring Developers Testmanager Test engineer IT operations department Competent department User stories, use cases, numbers for general usage and concurrent users 12.06.2017 www.proficom.de 13
Get the right people together Coordinating planning, test exection and analysis Developers IT operations department Project manager Testmanager Coordinating the whole development project. Creating test scripts and executes tests Get in touch before load testing for deriving test acceptance criteria and load models Name load critical functions. Receive test results. Fixing application Infrastructure information. What to monitor? Support monitoring Test engineer Competent department User stories, use cases, numbers for general usage and concurrent users 12.06.2017 www.proficom.de 14
Defining Goals Start with conceptual goals Jotting down goals that can t be measured will allow later filtering to create more focused goals Examples: Responsivness of a search function: Are we able to get search results within a reasonable time? System administrator s concern if the Update transaction functions during heaving usage. etc. 12.06.2017 www.proficom.de 15
Quantifying Goals Conceptual goal The search must be fast enough Confirm that save function works under heavy traffic Quantitive goal Search transaction time 5 sec for 2,000 concurrent users during peak hours (8-10 a.m.) Achive 300 concurrent users for save function at peak time (noon) 12.06.2017 www.proficom.de 16
Gathering system usage information Why? Allows to Decide which business processes to test Isolate peak loads and peak load times Document user actions and valid input data for each business process How! Using the site, first-hand Consulting with administrators (IT operations department) Consulting with executives (competent department) Researching competitor s sites 12.06.2017 www.proficom.de 17
Analyzing the system under test Why? Allows you to setup monitors Allows you to effectively coordinate with backend experts Provides system information that helps isolate performance problems How! Ask System Administrators Backend experts Application experts Database Administrators 12.06.2017 www.proficom.de 18
Identifying business processes to test Mission-Critical Business Process Business Processes that are crucial in day-to-day operations Heavy Throughput Heavy throughput business processes may not be mission-critical but are very popular Dynamic Content Dynamic content consists of sever requests that are customized for each user 12.06.2017 www.proficom.de 19
Identifying business transactions to test What are critical transactions (key transactions)? Transactions executed very often (heavy throughput) Transactions excetuted less often but creating a high system load Transactions executed less often but the response time should be measured anyway Transaction that are mission critical (in case of costs, public reputation, etc.) Finally, create (artifical) business processes from these transactions 12.06.2017 www.proficom.de 20
Identifying business transactions to test Mission critical transactions: E-commerce (add to cart, orders, payments,...) E-banking (transfers, trading,...) Get info from: Application experts Competent department Management teams 12.06.2017 www.proficom.de 21
Identifying business transactions to test Heavy throughput transactions: Homepage viewing Logins Site navigation Get info from: Competent department Server statistics 12.06.2017 www.proficom.de 22
Identifying business transactions to test Load intensive transactions: Searches Dynamic page generation Streaming media Get info from: Application experts Administartors IT operations department 12.06.2017 www.proficom.de 23
Business process matrix Business process Users (typical day) Users (peak time) Dynamic content Mission critical Test Sign in 70/hr 210/hr Low High? Creating new accounts Searches for artists View artist s page Purchase MP3s 10/hr 15/hr Moderate Low? 130/hr 180/hr Moderate Moderate? 20/hr 30/hr Moderate High 40/hr 90/hr High High? 12.06.2017 www.proficom.de 24
Defining a real world load Concurrency = Number of users acting (!) on the application at the same time (!) Application level How many users are active on the system? Business process level How many users are buying MP3s? Transaction level How many users press Buy MP3 now? 12.06.2017 www.proficom.de 25
Defining a real world load Do not focus on business processes/transactions or concurrent users alone! Better: How many users execute which transactions (business process) during which time? Dervive from business process matrix and statistics Create an extended transaction/business process matrix 12.06.2017 www.proficom.de 26
Time recordings saved Defining a real world load Example: Save time recordings 1.000 800 600 400 200 Hours 12.06.2017 www.proficom.de 27
Creating a transaction matrix 12.06.2017 www.proficom.de 28
Creating a transaction matrix 12.06.2017 www.proficom.de 29
Documenting user steps and input data 12.06.2017 www.proficom.de 30
Determining valid test data Sources: Application Data Data is resident in the application s database Examples: ID numbers and passwords User-Generated Data Originates with the user Examples: new unique ID or email address External Data Data is unknown before the application is run Examples: confirmation and purchase order numbers 12.06.2017 www.proficom.de 31
Analyzing the system Understanding system components Resources Web Server, Application Server, Database, Middleware, SAN, Load Balancer, Firewall Type Apache, IIS, WebLogic, Oracle, MS SQL, SAP PO, F5, PaloAlto Operating System Linux, Unix, Windows Server 2008/2012, Windows 7/10, Embedded 12.06.2017 www.proficom.de 32
Analyzing the system Mapping business processes to components Example: Searches -> Web Server, Application Server, Database, SAN Homepage -> Web Server Purchase orders -> Web Server, Application Server, Database, SAN 12.06.2017 www.proficom.de 33
Analyzing the system Mapping business processes to components Example: Searches -> Web Server, Application Server, Database, SAN Homepage -> Web Server Purchase orders -> Web Server, Application Server, Database, SAN Draw appropriate monitoring conclusions 12.06.2017 www.proficom.de 34
Monitoring system components Firewall not configured properly Load balancer not distributing properly Faulty Web server from cluster Database is slow or unable to feed data Application server cannot handle the request 12.06.2017 www.proficom.de 35
System performance monitors System resource monitors Network monitors Firewall monitors Webserver monitors Application server monitors Database monitors ERP/CRM monitors 12.06.2017 www.proficom.de 36
Test environment Mirrors the production environment Is dedicated to performance testing only (no interference with production users) Allows data to be written, read, destroyed Allows test system to be rebooted Allows to run the business processes correctly (application code freeze) Must contain sufficient hardware to generate the defined load 12.06.2017 www.proficom.de 37
Agenda 1. Why Load & perfomrance testing? 2. Stairway to load & performance testing 3. Choose the right tools 4. Execute and analyze the results 5. Summary 12.06.2017 www.proficom.de 38
LoadRunner on-premise Controller On-premise Load Generators Appplication under test 12.06.2017 www.proficom.de 40
LoadRunner on-premise + cloud LGs Cloud Load Generators Controller Appplication under test 12.06.2017 www.proficom.de 41
LoadRunner on-premise + cloud LGs Cloud Load Generators Controller On-premise Load Generators Appplication under test 12.06.2017 www.proficom.de 42
StormRunner + on-premise LGs StormRunner Cloud Load Generators On-premise Load Generators Appplication under test 12.06.2017 www.proficom.de 43
StormRunner StormRunner Cloud Load Generators Appplication under test 12.06.2017 www.proficom.de 44
Which solution should I choose? LoadRunner on premise LoadRunner + cloud LGs StormRunner More frequent testing Short term engagements Quick, high volume tests Determing baseline performance Remote Load generation Go-to platform for troubleshooting bottlenecks Web only Detailed reporting and analysis Limited reporting 12.06.2017 www.proficom.de 45
Which solution should I choose? HPE Loadrunner with ALM Integration LOADRUNNER ENVIRONMENT Load Generator APPLICATION QC / ALM Controller Virtual Users Web Server Application Server Database Server HTTP, SOAP, ORACLE, CITRIX ICA, FTP, SMTP, MAPI, LDAP, JAVA, Peoplesoft, SAP LASTTEST METRICS: CPU RAM IO MONITORING METRICS: CPU RAM DISK IO PROCESS 12.06.2017 www.proficom.de 46
Which solution should I choose? Integration HPE Performance Center PERFORMANCE CENTER ENVIRONMENT QC / ALM Performance Center Controller Load Generator Virtual Users APPLICATION Web Server Application Server Database Server HTTP, SOAP, ORACLE, CITRIX ICA, FTP, SMTP, MAPI, LDAP, JAVA, Peoplesoft, SAP LASTTEST METRICS: CPU RAM IO MONITORING METRICS: CPU RAM DISK IO PROCESS 12.06.2017 www.proficom.de 47
Which solution should I choose? Performance Center Very frequent (continous) testing All features like LoadRunner Easy test management within ALM Even more scalable Parallel and scheduled tests 12.06.2017 www.proficom.de 48
Agenda 1. Why Load & perfomrance testing? 2. Stairway to load & performance testing 3. Choose the right tools 4. Execute and analyze the results 5. Summary 12.06.2017 www.proficom.de 49
Phases of load and performance testing Planning Preparation Exection Analysis Defining goals Identify responsibles Identify testing capabilites of the application (PoC) Set-up test environment, Identify project time frame Defining monitors Configuring test environment Access test environment Creating test scripts Define load and iterations Creating test data Configure LoadRunner scenario Executing load tests Inform customer if test fails Documentation of findings and issues Documentation of results Validate test runs Central repository for documentation and results Report and recommendations Results presentation 12.06.2017 www.proficom.de 50
Test case density 1/s Phases of ideal load test Scaling Stability Overload 250 200 150 100 51 Scaling Test case density ~ load increases with number of users untill saturation of bottleneck device 50 0 0 100 200 300 400 500 User 12.06.2017 www.proficom.de 51
CPU load / % Phases of ideal load test 100 Scaling Stability Overload 90 80 70 60 50 40 30 20 10 Scaling Test case density ~ load increases with number of users untill saturation of bottleneck device 0 0 100 200 300 400 500 User 12.06.2017 www.proficom.de
Test case density 1/s Phases of ideal load test 250 Scaling Stability Overload 200 150 53 Stability Test case density ~ load = const. 100 50 0 0 100 200 300 400 500 User 12.06.2017 www.proficom.de 53
Response time / s Phases of ideal load test 20 Scaling Stability Overload 18 16 14 12 54 Stability Response time increases slightly 10 8 6 4 2 0 0 100 200 300 400 500 User 12.06.2017 www.proficom.de 54
Response time / s Phases of ideal load test 20 Scaling Stability Overload 18 16 14 12 55 Overload Response time acts impredictable 10 8 6 4 2 0 0 100 200 300 400 500 User 12.06.2017 www.proficom.de 55
Analyzing results Is there enough test hardware available in the test environment? Benchmark business processes against the testing hardware Take a business process and execute a small number of users against the application. Validates the functionality of the business process Potentially exposes unknown data dependencies Evaluate the testing infrastructure against the footprint. Do I have enough hardware to generate the user load? Do I have enough memory? Do I have enough CPUs? 12.06.2017 www.proficom.de 56
CPU Utilization [%] Transaction response time [s] Analyzing results Example: Response time and CPU consumption RT=f(Users) Step1 No. of VUsers Transaction Response Time CPU Utilization No. of VUsers CPU Utilization [%] 80 % 4 s 1 s 0 50 80 100 Step 2 No. of VUsers 0 10 % time 100 % CPU=f(Users) Testcase 50 % 10 % 0 50 80 110 No. of VUsers 12.06.2017 www.proficom.de 57
Analyzing results Don t forget to document the test execution and results Get togehter with application experts and administrators to analyze the findings and draw conclusions Derive recommendations for performance optimization Derive sizing recommendations Discuss the lessons-learned 12.06.2017 www.proficom.de 58
Agenda 1. Why Load & perfomrance testing? 2. Stairway to load & performance testing 3. Choose the right tools 4. Execute and analyze the results 5. Summary 12.06.2017 www.proficom.de 59
Summary Start early Consider performance as part of your application development Get the right people involved Define your goals (even non performance goals but project goals) Analyse your system to test, define load metrics and choose tooling Set-up monitors Execute the load tests Analyse the results and fix your application 12.06.2017 www.proficom.de 60
Questions?