IBM Global Services Addressing UNIX and NT server performance Key Topics Evaluating server performance Determining responsibilities and skills Resolving existing performance problems Assessing data for UNIX and NT platforms Planning for growth Despite planning and testing before big online events, Web site managers are being blindsided by traffic beyond their wildest expectations. Stacey Collett, The Glitch that Stole Christmas?, Computerworld, November 15, 1999. The use of Web technologies for commerce and communications has caused an economic revolution, impacting even the most sophisticated information technology (IT) professionals. This dramatic change challenges IT organizations to achieve a new level of server performance one that enables the business to grow and respond to new technology challenges. Implementing server performance management can help an IT organization effectively gain control of exponential growth and meet performance expectations for end users. Business results and server performance In the UNIX and Microsoft Windows NT environment, purchasing decisions and continued customer satisfaction depend heavily on server performance. If a Web application doesn t respond quickly enough, your customer may well abandon your site and go elsewhere. Reduced productivity and loss of business are just two obvious consequences of poor server performance. Another is that systems that fall short cannot provide the necessary flexibility to respond rapidly to the changing demands of the business. As you evaluate your current performance management capabilities, consider the following: What are my commitments to business units and what service level agreements (SLAs) are in place? Am I, as the service provider, legally bound to provide a response within a predefined time frame?
Adding hardware to address performance problems may provide relief for a short time. However, this action does not constitute a cost-effective approach to resource management. The performance analyst s responsibility is to understand system behavior and identify performance and availability exposures. The objective is to match system resources to application and user requirements. An ongoing challenge is dealing with requirements that continue to change during and after tuning is done. For example, a server may be called upon to support more users than was originally planned, or storage requirements may have grown because of changes in applications. Quickly identifying the need to add system resources or reprioritize workloads is an important skill of the analyst an iterative process of measurement, resource allocation and resource utilization. Assessment: The starting point When you experience server performance problems, a performance assessment may be necessary to identify probable causes. Certain indicators highlight the need for an assessment. For example, your users may be taking extended coffee breaks throughout the day because their tasks are not completing within the desired amount of time. Nightly backup or reporting jobs may still be executing when users arrive in the morning. Queries that had previously completed in less than one second are now taking minutes to execute. Clearly, assessing the root causes of these problems is essential. You also need to perform capacity planning assessments to avoid performance problems. Adding new technology or applications, mergers with another department or company, adding additional users or consolidating many servers into fewer systems all require an understanding of existing server performance and utilization. Whether you are addressing existing performance problems or planning for future changes, you need to have a thorough knowledge of your IT environment. A clear understanding of how to effectively utilize servers through system tuning helps you achieve user performance expectations. Defining responsibilities and skills When beginning the task of server performance analysis, you should first determine who will be responsible for the overall process, then confirm what technical skills will be required for the job. Do those skills currently exist in-house? If not, is there sufficient ongoing demand to invest in the education of one or more of your employees? Does the current performance situation allow adequate training time? Many companies have found that their time and dollars are best spent by outsourcing systems management. For some, that may involve monthly or quarterly assessments. Others may turn over all system-management tasks including performance analysis and tuning to an outside resource. 2
Forrester Research estimates that training will cost $6,000 to $8,000 per person annually and that a well-trained IT staff will spend about four weeks a year in training. Patricia Schnaidt, New Skills Prevent IT Brain Drain, Network Computing, October 30, 1999. Effective performance assessments follow a process of general guidelines for systems analysis and tuning. Although there is no book or guide that fits every situation, there are certain fundamental steps that should be taken to analyze and tune a system. Not all of these apply to every situation, and some suggestions may not be realistic for your environment. For instance, you may have diverse workloads that conflict with each other. Web servers present different tuning challenges than data servers or application servers. You need to balance resources based on the business priority. This includes users who want consistent acceptable response time, realtime business applications, and batch processes which run at predetermined times. As you begin to form your performance-management strategy, determine your major objective. Below are some common goals: Instantaneous response time for e-commerce or interactive users Optimized performance without the purchase of additional hardware Batch jobs completed within a specific time frame. Components of server performance management System managers with experience in keeping a computer system efficiently tuned recognize the following areas as essential for success: Resource monitoring Resource utilization must be monitored so performance problems can be easily detected either before they occur or immediately thereafter. Analysis and control Once a performance problem is suspected, the proper tools must be selected and applied so that the nature of the problem can be understood and the appropriate corrective action taken. Capacity planning Long-term capacity requirements must be analyzed so that sufficient resources can be acquired on a just-in-time (JIT) basis for maximum cost and utilization efficiency (i.e., will your system be ready for the next wave of Internet activity?). 3
Since performance management is an ongoing process, the initial goal should be both simple and quantifiable. Good performance or making the system run faster is not simple or quantifiable. Making a batch job run in five minutes and decreasing response time from two seconds to less than one second on a Web page are goals. By following a tuning methodology, you will learn whether or not your stated goals are attainable with current, available resources. You may find that you have conflicting or unachievable performance goals. That s why it s important to manage expectations during this goal-setting period. What you uncover can change the project s outcome, as well as your goals. Defined goals and objectives help you more effectively use your skills and tools. The monitoring method used should fit the server s place in your organization s overall IT strategy. A non-critical server may only need occasional monitoring using tools that come with the operating system or tools which are available in the public domain. Servers critical to your enterprise those which support key business functions may require constant monitoring with a suite of sophisticated tools that generate realtime statistical data. Factors that can impact your monitoring method decision are: How quickly does performance degradation need to be addressed? Who on your staff is experienced and available? What is your budget? Resolving existing performance problems The following flowchart and discussion provide a systematic approach to performance analysis and tuning for both UNIX and NT systems. System CPU If you determine that the system is CPU-bound, you have several options. Although a faster processor provides some relief for an under-powered system, it may not be economically feasible. It may be advantageous to spread the workload across two or more CPUs to obtain the required throughput and response times. Whenever possible, the resources should be used in parallel, rather than serial. This configuration can range from separate, standalone systems to tightly coupled parallel-processor systems. Spreading your databases over a number of servers on a network may also provide relief. 4
Performance tuning methodology CPU bottleneck? Yes Add CPU Reduce load Tune application No Memory bottleneck? No Disk bottleneck? Yes Add disk Reorganize data Delete data Yes No Add/tune memory Reduce load Tune application Network bottleneck? No Evaluate other factors Yes Add capacity Reduce load When upgrading from a single processor system to a multiprocessor (MP) system, make sure you either benchmark the application on the proposed hardware or obtain a reference account that is actually running the application in the environment under consideration. Some applications may actually run slower in an MP environment. Another way to reduce instantaneous CPU resource requirement is to schedule non-critical applications to run when the system is more lightly loaded. Running less important work in the background allows the more important foreground processes to have better access to the CPU. 5
Applications can also have a substantial impact on performance. If a user has access to source code and has a good working knowledge of the operating system and application, that user may be able to improve system performance dramatically by finding hot spots or inefficient code and making the necessary corrections. System memory If your problem is not related to CPU overload, the next step is to check available memory. Lack of sufficient real memory causes excessive paging or thrashing. This additional paging activity can cause the disk devices to be busy, resulting in more time spent waiting for I/O. If you cannot eliminate the thrashing through tuning, then consider installing more real memory. Even though most UNIX systems are known to consume most of real memory, this is not necessarily an indication of a memory constraint. If your system is constrained by memory, either tune the virtual memory management subsystem or add additional memory. Remember to take into account that some application designs have a significant impact on memory utilization. System disk utilization When you determine that memory is adequate, the next area to analyze is the use of disk resources. An unbalanced load on disk drives can cause one or more drives to be busy a large percentage of the time while others sit idle. This disparity often causes a bottleneck that throttles the I/O throughput and slows response times. The user should organize the data so that the normal access patterns equally utilize all available drives. If the logical volumes, file systems or files are fragmented, the time required for the disk to access the data may be much longer than would otherwise be necessary. For UNIX systems, one common gauge of disk utilization is the time the drive is active (%tm_act field of iostat). On a UNIX system, if time active exceeds 35 percent, the system may be I/O bound. On an NT system, if the drive activity exceeds 80 percent and the Current Disk Queue is greater than two, then the system may be I/O bound. Network utilization If you determine that system resources are not overutilized, your next step is to enlist the help of a network analyst. A detailed analysis of your network can identify design and traffic pattern problems that are causing poor performance. Your server performance may even be affected by problems that exist in unrelated areas of the network. 6
Assessment data for UNIX platforms When approaching a UNIX system performance problem, make use of these standard tools, which are available for performance monitoring: vmstat- Reports virtual memory, CPU and process scheduling statistics iostat- Reports CPU statistics and input/output statistics for TTY, disks and CD-ROMs ps - Shows current status of threads and processes sar - Provides CPU statistics on a per-processor basis, as well as numerous other statistics regarding system activity Public domain tools available from ftp sites include top, monitor and several others. The vmstat command reports statistics about processes, virtual memory, disks, faults and CPU activity. The command line syntax is similar to that of the iostat command. The most important columns in the vmstat reports are: r- Size of the run queue pi - Number of pages paged in from paging space po - Number of pages paged out to paging space us - Percent of user CPU time sy - Percent of system CPU time id - Percent of CPU idle time wa - Percent of CPU idle time while there is pending local disk I/O 7
Vmstat output will differ, depending on which UNIX platform you are using. These examples are from an AIX operating system. Even though the command line options may differ and the columns may have different headings, the same basic information is available from the various platforms. For iostat output, the most important fields to monitor are described as follows: %tm_act- Percent of time each physical volume is busy Kbps - Kbytes per second transfer rate Kb_read- Kbytes read during the interval Kb_wrtn- Kbytes written during the interval The %tm_act, Kb_read, and Kb_wrtn fields of the first report will give an indication of the overall disk load balancing. Iostat output can differ, depending on which UNIX platform you are using. Even though the command line options may vary and the columns may have different headings, the same basic information is available from the various platforms. Since iostat is relatively easy to use and interpret, it provides a sound basic analysis tool for the I/O subsystem. The utility is used for monitoring system I/O device loading. Reports generated provide valuable information to help in modifying system configurations. Making these modifications improves performance by balancing the I/O load between physical disks. Iostat is normally run for a predetermined number of iterations at a user-defined interval. The first report generated by the iostat command provides cumulative statistics from the time when the system was booted to the present. Each subsequent report covers the time since the previous report. This information is updated at regular intervals by the kernel. 8
Assessment data for NT platforms In the NT environment, make use of the Microsoft Windows NT Performance Monitor. This tool is included as part of the operating system, and contains enough functionality to perform a mid-level performance analysis of the operating system. The Performance Monitor provides a common graphical user interface to perform analysis on all major subsystems. This tool has three different views: chart, log, and report. The chart view provides live performance monitoring or the playback of a performance log file. The log view is the most important tool because it allows data to be saved and analyzed later. The report view is useful for obtaining averages of performance statistics over a monitoring period. There are three parts to NT server performance metrics: Objects- Memory, physical disk and processor are a few of the objects on NT. Counters- This refers to a statistic of an object, such as Pages/Sec, which is a memory-object statistic that refers to the number of pages read from or written to disk per second. Instances- Processors and disks are physical parts of the operating system that have multiple instances. If there are two processors on the system, then the processor object will have two instances CPU bottleneck Several important objects and counters should be observed when analyzing system CPU resources. The system object contains many important systemwide counters, such as Total Processor Time%. Microsoft suggests a CPU bottleneck may exist if the Total Processor Time% consistently is greater than 80 percent, and the Processor Queue Length Counter is consistently greater than two. To determine if the workload is being spread evenly across all of the processors of a multiple processor system, use the Processor Object s Processor Time%. The following graphic gives an example of this statistic. 9
Memory bottleneck To determine if a memory bottleneck exists, several key statistics should be measured. For example, Microsoft suggests that if the memory object s Pages/Sec is consistently over 20 Pages/Sec, then a memory bottleneck may exist. Disk bottleneck A disk bottleneck may exist if the Physical Disk Object s Disk Time% exceeds 80 percent and the Current Disk Queue Length consistently exceeds two. Planning for growth When approaching system performance from a proactive viewpoint, you should implement a method for viewing all of your server resources. Ideally, you want some automated indicators to show when resources are overutilized, well-utilized or underutilized. This will assist with your day-to-day and long-term planning. You may implement an off-the-shelf product that provides problem alerting as well as historical trending information. If your requirements are unique, you may choose to develop custom reporting using output from available system monitoring tools. If you are looking for assistance in interpreting system information and planning, one option is to outsource server trending and reporting. Whatever method you choose, having this information available can help you better utilize your existing resources and more effectively plan for growth. 10
Features to consider: Realtime alerts Diagnostic analysis Recommendations for improvement Graphical views Trending information Multiplatform support Summary As your server environment evolves, it is important to continually assess system resources. This paper has described how to effectively access the information you need to identify the causes of performance problems. Consolidating this information across the enterprise can help you plan for growth and change, and better leverage skills and resources. Although there is a cost of implementing server performance tuning whether you choose to train internal staff or outsource the cost of lost productivity, user dissatisfaction and the possibility of lost customers will far outweigh the expense. For more information For more information on IBM Performance Management and Capacity Planning Services, call 1-800-426-4682 (in the U.S.), 919-301-4141 (from outside the U.S.), or e-mail us at: capacity@us.ibm.com. 11
Copyright IBM Corporation 2000 IBM Global Services Route 100 Somers, NY 10589 U.S.A. Produced in the United States of America 06-00 All Rights Reserved IBM, AIX and the e-business logo are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. Microsoft, Windows and Windows NT are trademarks of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark in the United States and other countries, licensed exclusively through The Open Group. Other company, product and service names may be trademarks or service marks of others. References in this publication to IBM products and services do not imply that IBM intends to make them available in all countries in which IBM operates. IBM Integrated Technology Services organization in the United States, part of IBM Global Services, design and development of services offerings, has successfully achieved registration to the ISO 9001, 1994 international quality standard. G563-0338-00