IBM. Processor Migration Capacity Analysis in a Production Environment. Kathy Walsh. White Paper January 19, Version 1.2

Transcription

1 IBM Processor Migration Capacity Analysis in a Production Environment Kathy Walsh White Paper January 19, 2015 Version 1.2 IBM Corporation, 2015

2 Introduction Installations migrating to new processors often perform capacity assessments against workloads running on the migrated processor. These assessments are done to validate the expected capacity relationships and to ensure workloads running in the new environment are working as expected. The ability to do this analysis on a production system is made more difficult because of the varying nature of work on the system under review. Unlike a benchmark, where the load and test data are controlled and the system is configured to minimize change, production systems by their nature have significant variability. An analysis of any production system needs to be able to identify workloads where the inherent variability of the production system is reduced so comparative analysis can be done. In order to evaluate a production system it is necessary to apply some tests to determine workload repeatability and remove potential variability. Otherwise capacity information can appear to be out of bounds, when in fact, the differences can be due to a change in workload patterns. Multiple factors beyond just a new, faster processor can impact performance and throughput metrics. These factors include, but are not limited to: changes in the amount of processor storage application changes increase in the number of end users accessing the systems operating system and middleware maintenance changes in operating system levels exploitation of new features within the operating system, such as IRD or z/architecture significant changes in the size, and/or number of databases, or database buffering techniques use of dynamic SQL which changes access path based on power of the processors use of never ending work searches, a faster processor will search much more (and use more CPU) than a slower processor latent demand Other situations can also impact how workloads run on a newly installed processor. The most common situation is having a workload which previously was constrained by engine speed disrupting the allocation of CPU resources. Workloads which previously ran when the other workload was constrained now do not run as often causing issues in either performance or throughput. Changes in workload mixture can often be mistaken for under-performance of a new processor. This document will outline a methodology used to identify and assess relative processor capacity after a processor change for production environments. The methodology relies upon the application of statistical tests to try and identify stable and repeatable workloads in a production environment. However, as with any methodology, the rigor with which the analysis is done can impact the effort. This document will provide information regarding issues and conditions which can be encountered doing an analysis of this type. Statistical tests are used to identify workloads z Systems Performance IBM Corporation, 2015 Page 1

3 for further analysis, but as we will see, passing any given statistical test is only the first step in the analysis phase. Background Before we begin with the statistical tests we must first reflect upon the data used as a basis for the comparison. For z Systems processors this information is based on LSPR benchmarks. Information on LSPR is found at URL: https: www-304.ibm.com/servers/resourcelink/lib03060.nsf/pages/lsprindex?opendocument LSPR benchmarks provide information on a series of IBM benchmarks, run in a controlled environment, which are used to describe relative processor capacity. These benchmarks are used as the foundation in any capacity planning effort involving z Systems processors. Armed with the LSPR information the following steps need to be done to ensure appropriate capacity expectations are set for the processor migration which will be reviewed. First a workload needs to be developed for the LPARs involved in the migration. When choosing a workload there are two factors used to describe how well a workload should perform on a z Systems processor. These factors are the Level 1 Miss Percentage and the Relative Nest Intensity experienced by a workload. On a z Systems processor there is a hierarchy of hardware caches provided which are populated with data and instructions from central storage. The hardware caches reside in different locations within the microprocessor and have different sizes and performance capabilities. Generally, caches lower in the hierarchy are smaller and have better performance characteristics than caches at higher levels. On most current z Systems processors there are four levels of hardware caches; levels 1-2 resides on the core, level 3 on the chip and is shared by the cores on the chip and level 4 resides above the chip and is shared by all the cores on all of the chips. Caches residing above the core are referred to by an engineering term called the nest. When sourcing either data or instructions from higher levels of the cache hierarchy it will take more time, (processor cycles), than if the data is found in the level 1 cache. So understanding how often the requested information is not found in the level 1 cache, (called the Level 1 Miss Percentage (L1MP)), and where in the hardware cache heirarchy it is found directly influence the cycles per instruction required to run a workload. Workloads can be described based on how intensively they use the caches in the nest, hence a workload s relative nest intensity. The extent to which a workload relies upon caches in the nest or central storage will influence the number of cycles per instruction the workload will require. The LSPR data which is used to build capacity relationships among different z Systems processors is organized into 3 workload types based on a range of measured values for both the L1MP and the RNI. The workloads are called called LOW, AVERAGE, and HIGH. In order to do capacity planning for z Systems processors the workloads running on your current processors need to determine the L1MP and the RNI. The ability to measure and derive the L1MP and the RNI for a workload is provided by the CPU Measurement Facility, (CPU MF). Introduced with the IBM z10 processor this facility will z Systems Performance IBM Corporation, 2015 Page 2

4 provide the necessary data to determine the correct workload to do capacity planning. For more information on enabling the CPU MF facility please review the detailed planning information at URL: Once the workload has been established it is now necessary to determine the expected capacity of the before and after processor configurations. This step requires the analyst to identify the LPAR effects of both the before and after configurations. IBM provides a tool to help with this analysis called the IBM Processor Capacity Reference (zpcr). The zpcr tool is available as a free as is tool available by web downloading. To get the zpcr tool and information on training please see URL: One of the benfits of using the zpcr tool is it has the ability to read in the z/os CPU MF data to automatically determine the correct workload based on L1MP and RNI. Inputs to this phase would be SMF 70 and 113 records. In this phase it is important to use actual system configurations, and not configurations used in the capacity planning process. Many times the configuration used in the planning process are not the configuration actually deployed throughout the migration. It is imperative though for capacity expectations to be set using actual configurations and not planned configurations. This is especially true if the processor migration is across a lengthy time span. Each step where analysis is done needs to build capacity expectations based on the before and current configurations. So for instance, if the end game of a processor migration is to be a 16 CP processor, but a staged migration happens where the first LPARs migrated are only on a 10 CP processor the capacity expectations need to be rebuilt for the 10-way processor and not the 16-way processor. It is often at this point a checkpoint should be done to ensure all analysts involved in the process agree on the capacity expectations which will be used in the later data analysis. But once the capacity expectations are set, the next step is to begin the analysis phase. It is important to note the capacity expectations set in this phase are not used at all in the analysis phase. The analysis phase is going to provide an estimate of the relative capacity seen in the migration. The answer from the capacity analysis is compared against the expected capacity to determine how well the upgraded processor has met the capacity expectations. Data Analysis In order to control the variability which exists in a production environment the workloads are subjected to a series of statistical tests to see if they meet repeatability criteria. The approach used during the analysis is to look at the CPU used in relation to the I/O rate. The approach is based on the concept there exists a relationship between I/O and CPU usage. If I/O rates increase then it is expected to also see a comparable increase in CPU usage. It also is expected repeatable work would have the same CPU to I/O pattern each time the work runs. There are issues with this approach. First, the I/O patterns may, in fact, change between periods but within each period they are consistent, and so comparison between periods gives a false positive. Second, many workloads, especially online workloads, accessing a database manager z Systems Performance IBM Corporation, 2015 Page 3

5 will not execute a significant number of I/Os. Instead, depending upon the buffer hit ratios, the CPU demand of applications can increase due to increased workloads but if the locality of data reference is high in the database buffers additional database I/Os may not be needed and the I/O per CPU metric used in the analysis fails. In cases where database workloads are heavily used it is sometimes necessary to use transaction information to establish repeatability. In these cases it will be necessary to identify CPU per transaction code as a basis for analysis. Again there are the capabilities for false positives even at a transaction level. A transaction can be changed in such a way the code execution is consistent before the change, and consistent after the change, but change has occurred which impacts the CPU demand of the work. The identification of false positives from true conditions is more art form than science. It is also required in any analysis relying upon statistical tests to identify a condition. This document will discuss in more detail steps which need to be applied against the data to identify and manage false positives. The methodology followed in this phase of data analysis is to identify a set of repeatable work in a before period and a set of repeatable work in an after period. The before period best suited to this work is the period directly before the processor migration. The after period can be subject to more discretion. Generally the period where the majority of the migration is complete is best, though in a large server consolidation effort earlier analysis may be useful, say when a large or important LPAR has been migrated. B e fo re P ro c e s s o r A fte r P ro c e s s o r A ll J o b s R e p e a ta b le J o b s R e p e a ta b le J o b s A ll J o b s C o m m o n R e p e a ta b le J o b s D o th e a n a ly s is o n ly o n C o m m o n R e p e a ta b le J o b s Amount of Data Since statistical tests will be applied against the data it is crucial to have enough samples to make valid comparisons. Generally for batch and address space analysis this means at least 10 business days of data for both the before and after periods. Though analysis can be done with fewer samples the confidence in the data is reduced with fewer samples. Through experience we have not seen much more value added by having more than 10 days of data. Different data may be required if there are wide variations in workload throughout the period. For instance, application profiles can change dramatically based on month, quarter, or year end processing. It may be this change in profile which may lead the analyst to pick the same type of week in successive months rather than consecutive weeks. Inability to control this type of application z Systems Performance IBM Corporation, 2015 Page 4

6 profile can reduce the effectiveness of the analysis as large portions of the work may fail the repeatability tests. There are no hard and fast rules about picking representative periods on which to base the analysis. Instead the team doing the analysis needs to be aware of how different periods may affect the results and work as a team to provide the best possible input for the different periods. Any known issues with the periods chosen for review should be noted and documented as part of the analysis report. For online transaction data, such as CICS, IMS or Websphere, sufficient samples per transaction code are easy to obtain. For this type of data we recommend reviewing 1 hour of data from a peak time for 2-3 days for both the before and after period. This allows the analyst the ability to compare transaction codes for repeatability throughout the peak hour and across like hours on different days. Again the major goal of the analysis is to discover data which appears to be repeatable and stable enough in both periods to allow conclusions to be drawn regarding the relative capacity changes seen as a result of the processor migration. Because of the volume of transaction data available it may be very acceptable to identify certain activity levels which must be passed for inclusion in the analysis. An example of this would be transaction codes are included if they have more than 100 executions and/or use more than 5 seconds of CPU time. Any decisions made regarding which data is included in the analysis should be documented. Analysis Methodology As stated earlier we are going to build CPU to I/O relationships, or CPU per tran code relationships for work units within the before and after periods. In any given period work which is identified as the same, (same job name, step name, procname for address space analysis, or same SMFID, APPLID and transaction code for online analysis), will be examined for repeatability among the period s samples. Work which is not repeatable in the before period cannot be used as a basis for comparison to the after period. Likewise, work which is not repeatable in the after period cannot be used as a comparison to the before period. The CPU per I/O ratio or the CPU per tran ratio is computed for the appropriate period, and all samples are used to determine an average for the work item. The standard deviation of the work is computed, (a test for skewness), and a coefficiency of variance is computed, (standard deviation/average). Work units are accepted for inclusion in the appropriate period if the coefficiency of variance is 0.1 or less. Values higher than this indicate the work units within the same period are not repeatable and are excluded. For each time period the set of work units which pass the repeatability tests will be used to form a union of common work units which passed the repeatability tests. It is this union which will be reviewed to assess the relative processor capacity achieved as a result of the migration. z Systems Performance IBM Corporation, 2015 Page 5

7 For each before/after work unit identified a work throughput ratio (WTR) is calculated for the work unit. This is repeated for all work items which passed the previous repeatability tests in both periods. Two statistical metrics are created: Weighted Average. Weighted by the CPU contribution of the work unit. Median. Middle point of all of the WTRs The first test is to review the median WTR and the weighted average. If the work units used in the analysis are showing consistency among themselves then the median and the weighted average should be close. If they are not then some additional analysis may need to be done to understand why the values are different. Before we begin discussing steps necessary to ensure the validity of the data it is important to discuss what the expectations are for work units relative to the per CP capacity expectation set earlier. It is not expected for every work unit in the analysis to have the same WTR. In fact just the opposite is true. Work units will commonly be above and below the expected capacity. What we are looking for is on average the work is meeting the expectations. Handling Outliers in the Analysis When doing an analysis of this type it is necessary to review the data and identify and handle outliers. As previously discussed there can be false positives in the data; work which passes the statistical tests but doesn t pass the smell test. Results which are so obviously wrong they need to be removed from the result set. For example let s say the capacity expectation of a new processor would have a per CP WTR of 1.7. After running the analysis we find a work unit which has an WTR of 8.0. In this case it is unlikely the work ran this fast. In truth it is most likely the work which we tried to identify in the production system as the same was in fact not the same at all. The work met all of the statistical tests of a consistent pattern before and a consistent pattern after, but in reality the work unit in question is so far outside the expected range of performance it needs to be removed from the analysis. Additional investigation may be warranted to understand the job but it is unlikely the job is representative of the work running on the new processor. Again it needs to be remembered not every work unit will see the same WTR. Some variation is expected. In any distribution outliers will exist. The impacts of managing outliers needs to be done in a transparent manner. All outliers removed should be documented with the reason for the removal. But removal of outliers is considered normal and is a necessary step in an analysis of this type. Assessment The final test once the outliers have been removed is to compare the weighted average WTR against the per CP capacity expectations set in the earlier step. IBM has stated the capacity expectations which are set via zpcr and LSPR data have a confidence factor of +/- 5%. If the weighted average WTR is within 5% of the capacity expectation the processor is judged to be z Systems Performance IBM Corporation, 2015 Page 6

8 meeting expectations. Values off of this will require additional investigation. It may be necessary to examine the work in more detail to get a better idea of what is happening. There may be environmental factors or tuning factors which need to be investigated to better understand the work. These types of activities are outside the scope of this document. z Systems Performance IBM Corporation, 2015 Page 7

9 Appendix A Outlier Management Techniques Management of outliers and the function of trimming are actions which need to be applied against the result set. The following offers explanations of why certain trimming actions are appropriate to take. The goal of the analysis is to do as little trimming as possible. Weighted Average Different than the Median In general these two metrics should be close. What causes these two metrics to be different? Generally these situations arise with the following conditions: A work unit with a large amount of CPU whose WTR is out of sync with the rest of the work. A number of work units with a large number of executions whose WTR is out of sync with the rest of the work. Lots of work units which don t use a lot of CPU and have a WTR out of sync with the rest of the work. Simple Average vs the Weighted Average When using a simple average all work units in the analysis are judged to be equal contributors. A weighted average on the other hand gives more impact to work units which have more executions or use more CPU. Since this is a capacity exercise we are looking for work which drives the utilization of the processor. The utilization of the processor is more sensitive to larger jobs or jobs which have more executions. Lots of little jobs can influence a simple average but their impact on the utilization of the processor is negligible. WTR much different than expectations Certain work units are going to have WTRs much higher or lower than expected and can reasonably be explained. In many cases this work is demonstrating a false positive and should be evaluated for removal. This is a matter of judgment, there is no hard and fast rule. Result sets from application environments with unequal amounts of stable workloads Understanding the topology of the work which is being measured as part of this analysis is important. As stated before not all work will get the same WTR. We have also described why a weighted average is more important to use than a simple average. Namely a weighted average will better represent how the processor s capacity is being used. With this knowledge there are cases where the topology of the work influences the analysis. It is critical to understand a topology s influence and to look for cases where some additional conditions need to be examine before conclusions can be made about the relative performance of a new processor. The easiest method of demonstrating this condition is to build a sample environment. In this case we will use a CICS MRO environment made up of TORs, AORs, and FORs. In reality any application environment which has multiple components can exhibit this behavior but we use a CICS example because this type of environment is well understood. In order to highlight this discussion the following table will be used to represent the condition. z Systems Performance IBM Corporation, 2015 Page 8

10 We will discuss each row in the table and how it influences the analysis. Row CLASS A (AOR) CLASS B (FOR) CLASS C (TOR) CPU Seconds Description 2 77% 15% 8% Ratio of CPU Used by Class Benchmark WTR for each Class 4.77*1.9= *1.85= *1.7= ITRR weighted on USED CPU Seconds CPU Seconds - Repeatable Work 6 20% 2% 78% Ratio of repeatable seconds 7.2*1.9=.38.02*1.85= *1.7= ITRR weighted on Repeatable CPU Seconds In this environment there are three classes of work in the environment. As can be seen from row 1 the total CPU seconds used by the three classes equals 6500 seconds across some interval. The CPU seconds are not evenly distributed. Class A has either more work, or more complex work and represent 77% of the total CPU used. Class A is very much like an AOR where most of the transactions logic is executed. Class B is like a FOR where it is primarily I/O bound and the CPU used is primarily to manage data access. Class C is like a TOR where the function is primarily transaction management and routing, with very little logic execution. Row 2 has the ratio of work as judged by CPU seconds used by each class. Normally with a weighted average we would use row 1, CPU seconds, to drive the analysis. Clearly the work in Class A drives most of the capacity of the LPAR. Let s assume for right now we have total knowledge of the characteristics of this workload. Assume this workload was run in a controlled benchmark and we could directly measure the WTR for the work. The results of such a benchmark are listed in row 3. We have different WTRs for each of the three classes in the same environment. This is not unusual. Using this information and the CPU used for each class we can build a weighted WTR for the work using the ratio of CPU seconds by class. As such we get a WTR of 1.87 for the environment. This number is clearly being driven by the 1.9 WTR seen from the Class A workload. But production systems don t have the luxury of benchmarks to derive WTRs. Instead we need to apply statistical tests to see if we can identify the relative performance of work. So after applying the statistical tests for repeatability we get the number of CPU seconds which represents the repeatable workloads. This is documented in row 5. What you see is if you determine the ratio of repeatable seconds for the different classes of work the ratios are very different from the actual CPU seconds used in the environment. To further this example let s assume when we obtained the WTRs for the repeatable workloads the WTR s for the different classes matched the WTR s derived from the benchmarks (Row 3.) Using the information in row 6 it would now be possible to also build a weighted average based not on total CPU seconds but on repeatable CPU seconds. This is what is shown in row 7. In this z Systems Performance IBM Corporation, 2015 Page 9

11 case the overall WTR for the environment is being driven by Class C even though Class C represents only 8% of the entire capacity of the environment. Clearly two very different answers are derived, and both have merit depending upon the point of view. But why does it happen, and what do you do about it? Why Does it Happen? Experience has shown us two major reasons for why it happens: 1. Precision of the data 2. Trivial amount of CPU time Precision is the easiest to understand. If the most precision you have in the analysis is.0001 and the work is very trivial you will have a lot of work at the low end of the precision range. Any CPU per tran value, or I/O per CPU value from to will resolve to Work which isn t alike will look alike due to rounding. The second factor is CPU trivial work. The trivial nature of the CPU used in a job or transaction will cause work to fall into the precision issue mentioned above. It will also cause other issues on the processor. Processor performance is improved when cache and high speed buffer hit ratios are high. When very trivial work gets dispatched it needs to build it s working set, but just as the work gets started and can begin to see improved hit ratios the work ends. More complex transactions see more benefit, and hence better performance, because it can take advantage of the working sets it has built. This is why it is not expected for all work to get the same WTR as published LSPR ITRRs. Some workloads are more able to take advantage of the elements of the microprocessor design. But though some workload may see slightly less performance the impact is also negligible because the CPU content of the work is trivial. If the work became more complex then the work would begin to be able to take more advantage of the buffering and caching in the processor. Let s use the table we built above to demonstrate this point. Via the benchmark we know the Class 3 work has an WTR of 1.7. But as we do this work we publish an average WTR for the environment of The Class 3 work at 1.7 is 10% off of this published number. What does this translate to overall though. If we take 10% of the 500 seconds used by the Class 3 work this would be 50 seconds out of the 6500 seconds used. This would translate to approximately 0.8% of the CPU capacity used in the interval. The WTR is lower, but the amount of work represented is also correspondingly small, as is the impact on overall processor capacity. What To Do About It? Knowing it is happening is the first step. Understand much of this happens when the ratio of CPU seconds used becomes much different than the ratio of repeatable CPU seconds. Second, review the workload to see if they have precision issues, or are CPU trivial, or both. z Systems Performance IBM Corporation, 2015 Page 10

12 One approach to handling this type of situation is to assume the WTR created for the classes using the repeatable workload is a fair representation of the work running in the related class. Then using the WTR s derived from the repeatable workloads use as the weighting factors the CPU seconds used in the interval rather than the repeatable CPU seconds. Again, this type of outlier handling will not be needed if the ratio of CPU used seconds is close to the ratio of repeatable CPU seconds. z Systems Performance IBM Corporation, 2015 Page 11