IBM Software Group IBM Rational Software Quality Solutions - IBM Rational Performance Tester Denice Wong Technical Consultant Rational Software, IBM Hong Kong 2006 IBM Corporation
Agenda IBM Rational Software Quality Solution Overview IBM Rational Performance Testing Solution IBM Rational Performance Tester 2
Business Driven Development Software quality best practices Best Practices Focus on quality early Integrate quality into the life cycle Govern and control testing to ensure compliance Support testing of modular systems Leverage open standards 3
Focus on quality early Late defect discovery exponentially increases repair costs An IBM & Gartner rule of thumb for the relative costs to fix defects Lost revenue for each hour of downtime ranges from $89K to $4.5M 1 100 Application maintenance cost is 18.5% of total I/T spending 2 1 AC Market Analysis: Self-healing market June 2004 2 Source: Tivoli Marketplace Assessment, February 2004 Optimize Manage Model the Business Deploy Define Requirements Test 60 Analyze & Design Implement 1.0 10 1.5 4
Integrated quality capabilities ANALYZE DESIGN VALIDATION UNIT TESTING OPTIMIZE BUILD TESTING DEPLOY & MANAGE PERFORMANCE TEST FUNCTIONAL & SYSTEM TEST 5
Govern and control testing ANALYZE DESIGN VALIDATION UNIT TESTING OPTIMIZE GOVERN TEST LIFE- CYCLE BUILD TESTING DEPLOY & MANAGE PERFORMANCE TEST FUNCTIONAL & SYSTEM TEST 6
Rational Software Development Platform GOVERNANCE DASHBOARD Solutions for geographically distributed development, compliance, SOA Process & portfolio management IBM Rational Portfolio Manager IBM Rational Method Composer Best practices content (IBM Rational Unified Process IBM Tivoli Unified Process, Portfolio Management) Requirements & analysis IBM WebSphere Business Modeler IBM Rational RequisitePro IBM Rational Software Architect IBM Rational Software Modeler IBM Rational Rose Data Modeler Design & construction IBM Rational Software Architect IBM Rational Software Modeler IBM Rational Data Architect IBM Rational Application Developer IBM Rational Web Developer Software quality IBM Rational Performance Tester IBM Rational Functional Tester IBM Rational Manual Tester IBM Rational PurifyPlus IBM Rational Robot IBM Test RealTime Change & configuration management IBM Rational ClearCase IBM Rational ClearQuest IBM Rational Team Unifying Platform IBM Tivoli Provisioning Manager IBM Tivoli Configuration Manager IBM Tivoli Intelligent Orchestrator Partner ecosystem & open computing: Eclipse, Linux, Microsoft Windows, UNIX, IBM z/os 7
Rational Software Quality Solutions BUSINESS SOFTWARE QUALITY SOLUTIONS DEVELOPMENT Requirements Test Change Defects Capture requirements Document use cases Monitor change Developer Test Run-time analysis Profile memory Code coverage Component & services testing Test and Change Management Govern testing Trace to business needs Audit progress/change Monitor quality Automated Automated test creation Find regression defects Test Automation Functional Test Manage code change Associate with builds Deploy to servers Manual Early business process tests Modular components Quality Metrics Security Test AppScan WebXM Track defects to resolution Trace to business needs Performance Test Model users Simulate workload Find bottlenecks Pinpoint production problems Project Dashboards Detailed Test Results Quality Reports OPERATOINS 8
Rational Software Quality Solutions BUSINESS SOFTWARE QUALITY SOLUTIONS Test and Change Management Requirements Test Change Rational RequisitePro Rational ClearQuest Rational ClearQuest Defects Rational ClearQuest DEVELOPMENT Developer Test Rational PurifyPlus Rational Test RealTime Functional Test Test Automation Rational Functional Tester Plus Automated Manual Rational Rational Functional Tester Manual Tester Security and Compliance Test AppScan WebXM Performance Test Rational Performance Tester OPERATOINS Rational Robot Quality Metrics Project Dashboards Detailed Test Results Quality Reports 9
Agenda IBM Rational Software Quality Solution Overview IBM Rational Performance Testing Solution IBM Rational Performance Tester 10
What Is Automated Performance Testing? The process of exercising an application by emulating actual users with a load generation tool for the purpose of finding system bottlenecks Controller System Under Test 11
Why do Performance Testing? Because a break at any point in your system means your customers are not getting the service you think they are System Under Test 12
Challenges of Performance Testing Complex Tools & Complex Task Tool complexity and lack of experience intimidates many first time users We don t have a lot of experience building good load tests. Most load test tools are much more complicated than we require. 13
Challenges of Performance Testing Complex Tools & Complex Task Tool complexity and lack of experience intimidates many first time users We don t have a good understanding of what is causing the problem Testing Tools Lack Insight Tool can find the problem, but not diagnose the root cause I need to know more than simply where a problem is occurring. 14
Challenges of Performance Testing Complex Tools & Complex Task Tool complexity and lack of experience intimidates many first time users Our developers have no way of performing load tests. Testing Tools Lack Insight Tool can find the problem, but not diagnose the root cause Hidden Costs Hardware intensive tools and inability to share information with team mates increases cost of testing We don t have enough hardware to run our tests. 15
Performance Testing with IBM Rational Performance Tester Test automation for the novice and the professional IBM Rational Performance Tester Performance Testing for Web Based Applications Key Benefits Built for Day 1 Productivity Mask complexity to get the job done Advanced Data Access & Manipulation Automated data variation and synchronization Root Cause Analysis Identifies location and root cause of performance problem in hardware and software 16
Creating a Performance Test Creating a performance test is a three step process Build Scripts Script Creation Considerations Visual test editor, varying input data & correlating server responses 17
No-Code Tests Tests are represented in a tree view as a list of web pages and their elements, such as images. No programming necessary to create a test Underlying detail is accessible but hidden to simplify test creation and maintenance 18
Visual Test Editor Test Organized as tree view of access pages. No code Test Navigator Access to Tests and Schedules Additional Info View detailed info about requests and responses Organization Top level branches are pages. First branch under the page is the html container. Under that are all the page elements, such as images, cascading style sheets or java scripts 19
Visual Test Editor Enhance tests without programming Loops Conditions Verification Points Validate server response code Validate server response size Tests not Scripts Power User Features Manually add HTTP requests Add custom Java code With Atlantic, we are no longer using the word script, instead, the tool now creates tests. 20
Creating a Performance Test Building Scripts Session No. 100 User Name: JSmith Session No. 101 User Name: CBryson Session No. 102 User Name: TJones Automated creation of parameterized tests Literal values used during recording are replaced with variables At playback time, variables are populated with data from a datapool Data for playback can be imported from external data sources No user coding involved 21
Automatic Identification of Variable Data Automatic server response correlation Items in purple indicate that RPT has automatically correlated a piece of data unique to this run Data correlation The automatic process of flagging and parameterizing data that is unique for every test run. In the example on the left three values have been correlated: custid, orderid and SessionID. Lets focus on orderid: orderid is a unique order number that is generated for every purchase This value cannot be hardcoded in the test and is not. RPT will automatically use the order ID that is generated by the web site for every playback. 22
Built-in Datapool Editor for Varying Test Data RPT Automatically identifies candidates for datapooling All data entered by the user is a candidate for datapooling The next 6 slides walk through the process of datapooling values 23
Creating a Performance Test Creating a performance test is a three step process Build Scripts Schedule Workload Script Creation Considerations Visual test editor, varying input data & correlating server responses Scheduling Considerations Accurately representing a true user workload 24
LoadSmart scheduling Easy to use Model complex real world scenarios Point and click scheduling 25
Creating a Performance Test Schedule Workload Powerful and flexible scheduling Visual schedule editor for no code scheduling Coordinate timing and dependencies Accurately model real users workloads Dynamically increase the load during the run 26
Creating a Performance Test Creating a performance test is a three step process Build Scripts Schedule Workload Execute & Analyze Script Creation Considerations Visual test editor, varying input data & correlating server responses Scheduling Considerations Accurately representing a true user workload Execute and Analyze Considerations Validating responses & finding the bottleneck 27
Root Cause Analysis Root Cause Analysis is a collection of features which provide the user with information to diagnose the cause of their performance problems Root Cause Analysis collectively refers to the following functionality Problem Analysis Tools Technology formerly bundled as the IBM Performance Optimization Toolkit Response Time Breakdown Agentless Resource Monitoring Report Data Overlay Report Filtering by Time 28
Performance Problem Identification During Test Page Performance Report Shows average response time per page. Highest bar = Performance Problem Rational Performance Tester easily identifies bottlenecks Next logical question is Why? Root Cause Analysis provides to tools to answer this question 29
Performance Problem Identification During Test Rational Performance Tester easily identifies bottlenecks Next logical question is Why? Root Cause Analysis provides to tools to answer this question 30
Live Response Data Collection Problem Diagnostic data capture with the Data Collection Infrastructure Primary Use Case: Performance test is run with problem analysis features enabled capturing live diagnostic data via DCI Data collection infrastructure Pre-instruments WebSphere and Weblogic servers to capture class and method level information Is run on all tiers of the application Uses Application Response Measurement (ARM) technology and Java JVMPI Profiler Interface to capture data Configuration Process Install on all tiers of application (Windows, Linux Redhat/SuSE only) Run Application Server Instrumenter to instrument WebSphere and WebLogic Servers Start DCI via Start Menu Communicates over port 10002 Application Server Instrumenter GUI Interface to ARM instrument WebSphere and Weblogic servers 31
Problem Analysis Tools On schedule, specify tests for which to capture additional data On schedule, specify tests for which to capture data Specify detail level and sample size Best Practice: Minimize superfluous data Capture problem diagnosis data only for relevant pages 32
Problem Analysis Tools Page Level Diagnostics Feature: Response Time Breakdown can be done at schedule level or at page level Benefit: Page level granularity enables you to filter out irrelevant information to your investigation On test, specify pages for which to capture additional data 33
Response Time Breakdown Feature: Breakdown page response times into composite element response time Benefit: Response data broken down by tier and by transaction component (JDBC, JSP, Servlet, etc ) Breakdown data drills down to highlight slowest components of a given page. Breakdown into tiers and components is a unique competitive advantage for RPT. 34
Deep Diagnostic Data Providing the details you need to solve the performance problem mystery Execution Statistics Shows response time for every method UML Sequence diagram showing method calls between classes with timing information. User can drill down to source code. 35
Resouce Monitoring Configuration on Schedule Agentless monitoring of local or remote Windows and Unix machines Unix requires rstatd daemon to be running Windows requires File Sharing enabled and a connection established with an account with Administrator access Minimum requirement is remote registry monitoring 36
Agentless Resource Monitoring Feature During a performance test run, RPT can capture resource monitoring information from IBM Tivoli Monitoring UNIX rstatd monitor Windows Performance Monitor Does not require any installation on any tier of the application Subset of default counters selected to monitor key resources Benefit Provides additional data during results analysis to identify root cause of performance problems 37
Resource Monitoring Data Incorporate resource data into response data Tech Tip Ensure time clocks are synchronized across machines 38
Performance & Resource Statistic Report Overlay Data from different graphs can easily be consolidated on to one graph. Includes resource monitoring data 39
Report Filtering by Time Range Feature: Filter results on any report to focus on a given time range Benefit: For long runs, enables focused analysis on subset of data most interesting to customer 40
Ensure Application Scalability and Performance IBM Rational Performance Tester Extension for Siebel Improve capacity planning which can lead to lower hardware costs Simulate multiple concurrent user types and business transactions Generate work-loads that stress all points in the architecture Web Server Isolate the source of performance bottlenecks and optimize resource utilization before deployment Siebel Validation ensures technical readiness OLTP OLTP OLTP 41
Ensure Application Scalability and Performance IBM Rational Performance Tester Extension for Citrix Presentation Server Reduce the cost and time associated with performance testing with an easy to use, "nocode" test recording and execution solution Test system scalability and performance by emulating users with varying business transactions, usage patterns and work-loads Citrix Presentation Server Perform pre-deployment capacity planning tests to ensure efficient allocation of server resources 42
Performance Testing of SAP Applications The process of testing SAP applications Follows the same steps as testing web based applications Record Edit Execute & measure response times But requires new technology to enable the process SAP Recorder SAP Protocol Browser for test editing SAP Performance Reports SAP HTTP Interface now supported SAP 4.6, 4.7 and 5.0 offer an HTTP interface to SAP. Visual Design is a feature of SAPGUI which allow the customer to modify the colors & fonts of SAPGUI 43
Service Quality Management Functional and Performance Testing of Web Services from a common interface Rational Tester for SOA Quality Automated regression and functional testing for GUI-less Web services Rational Performance Tester Extension for SOA Quality Performance Testing for Web Service based applications 44
45