Identifying Application Performance Risk

Transcription

1 Identifying Application Performance Risk Performance Engineering COLLABORATIVE WHITEPAPER SERIES

2 COLLABORATIVE WHITE PAPER SERIES: Identifying Application Performance Risk Under what conditions will an application require a performance test? Not all applications require performance testing, and the same application may not require repeated performance testing for every release. The selection process for performance testing frequency must consider user population, application type, technology, changes to features and function, and how nonfunctional requirements are monitored in the software development life cycle (SDLC). I. Introduction A typical enterprise has thousands of applications. In a business unit, there can be several hundred applications. As the business continues to evolve and change, so will the applications evolve and change to support the business needs. The changes made to the application may increase the risk to performance, scalability, or stability. Depending on the business s tolerance for disruption, these changes may require some level of performance and scalability testing to verify the application can still process the accepted business volumes while staying within service level agreements. These application changes are typically scheduled on a release calendar. Customer facing applications should be considered critical to the business, while internal facing applications may be less critical. However, what if the internal application is supporting the executive level of the company and provides key information for decision-making? Some enterprises believe inherently that performance and scalability testing are required, while others may leave the decision to the business units. Enabling the enterprise for performance engineering requires an investment. This investment is considered to be an indirect investment, as it may not be immediately linked to a revenue generating process. The value of the investment in performance engineering can be quantified. For example, when performance engineering recommends design changes allowing a revenuegenerating website to process more orders, the return on investment is clear. In a large enterprise, performance engineering resources are often in short supply, and budgets are generally under constant pressure. This leads IT and business decisionmakers to ask certain questions, such as: What guidelines can you use to help allocate these resources across the project portfolio? How can you make sure you have not missed an application that required performance testing, and how can you make sure you re not testing the wrong applications? Are you sure the results from performance are accurate and allow management to make informed decisions? 2

3 II. What factors should you consider when selecting an application for performance testing? Not all applications require performance testing, and the same application may not require repeated performance testing for every release. The risk factors you should use to evaluate your application include:. User population The people who use your application are critical to the decision. Questions may include: Who are the users of your applications, how many concurrent users are there, and is the number of users increasing? Are the users purchasing products or services from your business? Are they external or internal users? How easy is to for your user base to switch to a competitor if you website is not performing well? How is the application accessed? Access types may include browsers (both desktop and mobile), call center applications (both client server, legacy, and web), and even internal voice recognition (IVR) clients. The amount of transactions for each access type and application must be considered when defining the workload for the user population. 2. Application type The application type itself can dominate the risk factors. For instance, questions to ask may include: Is the application an online retail website? Is there both a desktop and mobile website/mobile app? Typically a revenue generating website requires a performance test for every release. Is the application a key component in the enterprise architecture that other applications use? If so, this is deemed a critical application and may require performance testing for every release. Is the application a batch process with a strict window of processing? How critical is this application to the business, and how is it rated? 3. Application technology The state of the application technology stack can be a significant risk factor. Generally, the technology platform does not change from release to release. If the underlying technology is stable and is well known to the application development team, a performance test might not be required. However, if a new technology is being introduced or replacing one of the tiers, there may be a great risk and thus testing is required. Likewise, a significant upgrade to a vendor product could warrant performance testing. It is important to consider the scope and impact of changes to any key components. 4. Application features and functions The amount of modified code or new code in an application can create new performance risks. Understanding the impact of the changes is critical to determining if performance testing is required. Potential analysis questions may include: How has the new or modified business feature changed the behavior of the application? Were the changes extensive and across the client, application services, and database? What percentage of the code was impacted by the new or modified services? 5. Software development process Analysts may consider questions such as: Does the SDLC track non-functional requirements during the lifecycle? How are those non-functional requirements communicated from the requirements, design, development, testing and deployment teams? Have key business transactions or services been identified with stringent response time requirements, or strict throughput requirements? What architectural risk analysis, prototyping, or other types of testing have been done throughout the lifecycle that may mitigate the need for formal performance and scalability testing efforts? 6. Production issues with the last release Recent history can be an indicator for the future. If the application went into production and the last release had performance, scalability, or stability issues, then it may require a closer look at the application to determine if the issues have been truly mitigated. Otherwise, performance testing is required. Similarly, resource utilization patterns and trends may be used to assess the need for further risk mitigation. 3

4 COLLABORATIVE WHITE PAPER SERIES: Identifying Application Performance Risk 7. The schedule of performance tests Applications will undergo performance testing at different times during their lifetime. The application can be tested before it is ever released into production; performance testing can be scheduled for every major release; or performance testing can be scheduled based on the extent of the application s changes. III. Risk factors elaborated. User population The key information to capture is: How many people use the application on average and during peak time? Who uses this application? What is the behavior of the users workload? In an analysis, we may ask: What would be the business impact if this application exhibited performance problems? Is your business in a highly competitive market, where dissatisfied users can switch to you competitor? For instance, if the response time doubled or tripled from normal, how would the user experience degrade, and what would be the financial impact of service calls and shopping cart abandonment? How long would this condition be tolerated and under what conditions would it be tolerated? The matrices shown below may be used to identify and assess potential performance engineering risks, using a simple -2-3 scoring system: User Population Characteristics Figure : Figure title Business internal: Senior management or board level, enterprise-wide Business external Consumers Average number of concurrent users Peak number of concurrent users relative to normal business load based on month-end or periodic event that occurs regularly Rate of user population growth Flash or sudden spikes in concurrent users, above and beyond your current peaking factor: Based on a new business model of broadcasting messages to your client base telling them to all come to your website Currently a novelty event for many businesses, not accounted for in the system design Eventually folded into the traditional peaking factor 2 2 : Thousands 2: Hundreds Risk Level : requires a performance check 2: depends on other extenuating factors 3: does not require a performance test : Application peak TPS is more than five times the typical TPS for the business : If population growth doubles in less than nine months : A significant increase is expected due to the business winning a new contract or large client 3: No user population growth : Upcoming flash or spike events will add to existing application workload 4

5 If you have a large number of consumers that is growing, you would require performance and scalability testing. For example, flash events are a key risk area because they are occurring more frequently for businesses on application platforms that were neither designed nor tested for sudden workload bursts. 2. Application type Most businesses will rate their applications based on criticality to the business. The rating is typically for availability and recovery time. Ratings can range from high availability (almost no down time) rated A, to minutes of down time rated B, to hours rated C, to days rated D. These ratings factor into your selection criteria. Should a C rated application be performance tested? Under specific conditions it should; for instance, it is a major application upgrade or if it is chronically slow for the users to the point that it is interfering with their workflow and productivity. Application Characteristics Online website retailing (may include both desktop browser website and mobile website/app) Financial transactions (brokerage, 40K, etc.) Corporate support Customer portal: This is where the customer can maintain their account information, view their history of transactions, or check the status of an order Business portal: This is where the business partner can maintain their account information, view their history of transactions, or check the status of an order. It could be for an insurance agency with a large number of independent agents Business intelligence and analytics: Large volumes of data analysis, tight reporting deadlines or regulatory filings Date hub or integration hub: A key component where many applications are dependent upon Informational or marketing: User experience is important if this is for consumers Batch processing: Constrained processing windows Risk Level : requires a performance check 2: depends on other extenuating factors 3: does not require a performance test : Enterprise-wide use 2 or 3: Group-wide : Open enrollment is a key business process that occurs once per year : Most customer facing systems require some level of performance testing, the enduser response time is critical to how your customers view your business : The business partner is responsible for driving revenue to your business 2: The business partner provides a service to your business that is part of a workflow : Service level penalties if late : First mover advantage, business, competitive edge 2 or 3: Weekly, monthly, quarterly reporting 2 2 or 3 Rating depends on extent of penalties for violating SLA s 5

6 COLLABORATIVE WHITE PAPER SERIES: Identifying Application Performance Risk Document management Imaging systems: These are typically scanning systems that are used in mail order processing applications, or where signature is required. They are part of an essential workflow A core Enterprise-wide component, used by many other applications : Timeliness of documents is critically important, e.g. news agency or day-trading investment advisor 2 or 3: Otherwise : Part of a time critical workflow. The documents must be scanned quickly and made available quickly or 2 3. Application technology When you are using unfamiliar technologies, you need proactive performance management. In general, the application technology stack does not change for every release. Nevertheless, it is important to consider the cumulative effect of small changes over time with respect to online features and batch processes. Technology Characteristics Risk Level : requires a performance check 2: depends on other extenuating factors 3: does not require a performance test Well-known and stable technology stack Accessed via more than one client presentation tier (i.e., classic browser website, mobile website, web app) New technology for a key component: For instance replacing the WebSphere application server with Jboss, or replacing a C++ batch process with Java batch process Significant technology upgrade: For instance, upgrading the WebSphere application server, upgrading the Database to a new major release. The key here is to identify impact and determine the proper level of testing Significant application upgrade: This is typically for a third party vendor who made significant architecture changes, particularly if the new version of the application has not been demonstrated at full business volume New component added to the application: The application has new component that will become part of most of the key business transactions 3 or 2: It is critical to understand the number of transactions that use this component and the nature of the transactions 6

7 4. Application features and functions This section considers the magnitude of the changes made to the application for this release cycle. The goal is to assess the risk the new change poses to the performance, stability, and scalability of the application. Where new database tables were added or existing ones modified, did this impact the database partitioning scheme? On the application server, were new services introduced, or existing ones modified? Will this impact the application server s scalability profile? Did the web client change to introduce asynchronous web server requests? How will this impact the transaction arrival rate to the web server and application servers? Has this application experienced performance or stability issues in the recent past? Is there a history of performance issues after a new release? Is the application sensitive to a sudden increase in user load or transactions per second? Under what conditions does this occur? Impact Risk Level : requires a performance check 2: depends on other extenuating factors 3: does not require a performance test Web client Web Server Application server Database server or 2: Introducing asynchronous Web server requests : New plug-ins or new connections to application servers : Services added or changed with suspected performance impact : Impact to database partitioning scheme /2: New tables added to high volume transaction : New requirements for keeping and accessing historical information All tiers impacted by change Percentage of overall code changes or new code developed. For instance, if there are 00,000 lines of code, what percentage was modified? Batch processing where new jobs were introduced, was the concurrency model impacted? Reporting server : More than 5% 2: 5%-to 5% : Concurrency model impacted : New tools providing ad hoc and wider or deeper queries 7

8 COLLABORATIVE WHITE PAPER SERIES: Identifying Application Performance Risk 5. Software development process The software development process should identify non-functional requirements, such as performance, scalability, maintainability, usability, and others. During the requirements phase, any key business use cases that have response time requirements must be identified and tracked through the process. Also, key business use cases for batch process throughput should be identified. In general, activities that support performance engineering should be conducted throughout the entire lifecycle, and they should be defined in a standards document so that assessment of performance risk can be understood. Coordinated Performance tests: The performance team will lay out a plan to coordinate the goals and test cases for performance in architecture proof-of-concepts (POCs), unit testing, functional quality assurance (QA), and performance testing. The development/build phase has a unit testing activity, the functional QA testing phase will have a performance test activity, and the performance testing process will require the outcome of those two activities. Early in the lifecycle, an architectural POC may be required, and it should include a set of performance, stability, and scalability tests. The performance team can unify the strategy of introducing performance testing into the other testing activities. The performance team can introduce micro performance tests into the other phases of the SDLC where they already have testing activities, for architecture/design, development, and QA. 6. Recent release and production issues The recent past can be an indicator of the future. If performance, scalability, or stability issues arose from the last release of the application, performance testing is needed on the next release. In some instances, the application development team may have a poor track record for managing application performance, resulting in frequent performance issues. The performance team must see the application performance management reports from the production operations team. This will allow them to determine the nature of the production issues related to performance. It will allow them to build a historical profile for the production behavior of the application. This linkage of the performance team to the operations team provides a feedback loop to the performance team. This information will allow the team to adjust the performance test scenarios based on changing workloads and new or modified business functions. 7. The schedule of performance tests Applications will undergo performance testing at different times during their lifetime. The application can be tested before it is ever released into production; performance testing can be scheduled for every major release; or performance testing can be scheduled based on the extent of the application s changes. a. Project-based schedule: Project-based performance tests can be driven by the nature of the change to the application and technical architecture, rather than by the profile of the user population. This approach is appropriate if the user population is not growing, but the technology is changing. If the application uses a vendor product as a key part of the workflow, and the vendor has introduced a new major release, the level of risk may be high. Before agreeing to forgo in-house performance testing, the application owners must be more than satisfied that the vendor had done proper performance testing. Once the new version has been verified not to harm the response time or throughput, the performance test project is over. Oftentimes, the application is custom-developed by the IT organization for the business. The custom-developed application may be in need of technology upgrade or replacement. This would require a project-based approach to performance testing, possibly coupled with a releasebased approach to the schedule. The custom-developed application may be scheduled for iterative deployment across different release cycles. The risk of the new technology or upgrade must be understood. What did the vendor or IT development team change? You must understand how the changes impact the users workflow. You must measure the before and 8

9 after performance of the system. In order to do a proper comparison, you must understand the key business transactions response times and the batch process throughput of the existing version. b. Release-based schedule Release-based tests are recurring and are scheduled for every release. Since the application has been deemed critical, it requires a baseline test, performance validation, or performance characterization for every release. The type of test may change from release to release due to the nature of the change. The test scenarios can change based on observations from the production environment. Likewise, a changing production workload or changing user behavior will require the performance test cases to change. The performance team must have a link to the production application-monitoring and capacity-planning team. The user population could be increasing, and the business may have introduced new features or functions that have changed since the last baseline test. c. Production support schedule Production support test cases are required to help triage production issues. When issues arise due to performance, scalability, or stability, the root-cause must be found, and one or more solutions must be defined. These tests are disruptive to the planned performance testing schedule and environment. Oftentimes, the performance team has the only properly sized environment to reproduce the production issue. Release and configuration management become critical in this situation. You need mature processes to be able to suspend the current performance testing schedule and execute on the production issues. These test cases are created on-demand and adjusted as new information is uncovered. The tests are created to either locate the root-cause or to help define a solution. IV. Conclusion This approach will allow you to allocate the right resources to the right projects and applications. Performance engineering and performance testing should not be done in isolation or behind closed doors. The selection process and the risks must be visible, and there must be communication between the performance team and the business, ultimately to answer the questions of why are we testing? and what are we getting from those tests? Establish a transparent selection process, as some application owners budget for performance testing and others do not. This selection process will allow you to work with the applications owners who do not budget for performance testing, providing them a structured process for identifying risks to the business as a basis for selecting the applications to include in performance testing. Is the current vendor telling your business they must upgrade? Often software products are selected by the business because they provide a set of key business functions, while little attention is paid to the non-functional requirements of the business. This process will allow you to assess the risk introduced by a vendor solution, either during the selection process or with an established vendor who wants you to upgrade to the new version. This approach provides the criteria for defining the performance risk or scalability risk the application may or may not present to the business. For instance, you should always be looking out for business initiatives that introduce a new extreme workload, such as a new marketing campaign that will encourage customers to login to see the new product. Then in response, you must design and execute a scalability test to verify the behavior of the application under the anticipated flash-type workload. Most importantly, this approach provides visibility into the performance testing risk-based selection process. This approach allows you to communicate the risk to business and other key stakeholders by showing the connection between the business risk and the testing plan to reduce that risk. Now that you have identified the applications that should be performance tested, what types of tests should you be designing and executing? See the next white paper from Collaborative Consulting on the different categories of performance tests. 9

10 COLLABORATIVE WHITE PAPER SERIES: Identifying Application Performance Risk Collaborative Consulting is a leading information technology services firm dedicated to helping our clients achieve business advantage through the use of strategy and technology. We deliver a comprehensive set of solutions across multiple industries, with a focus on business process and program management, information management, software solutions, and software performance and quality. We also have a set of offerings specific to the life sciences and financial services industries. Our unique model offers both onsite management and IT consulting as well as U.S.-based remote solution delivery. To learn more about Collaborative, please visit our website at us at sales@collaborative.com, or contact us at Copyright 204 Collaborative Consulting, LLC. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. WP.24.06