Designing for Performance Management of Mission Critical Software Systems in Production

Transcription

1 Designing for Performance Management of Mission Critical Software Systems in Production Raghu Ramakrishnan TCS A61-A, Sector 63 Noida, Uttar Pradesh, India Arvinder Kaur USICT, GGSIPU Sector 16C Dwarka, Delhi, India Gopal Sharma TCS A61-A, Sector 63 Noida, Uttar Pradesh, India Traditionally, the performance management of software systems in production has been a reactive exercise, often carried out after a performance bottleneck has surfaced or a severe disruption of service has occurred. In many such scenarios, the reason behind such behavior is never correctly identified primarily due to the absence of accurate information. The absence of historical records related to system performance also limits development of models and baselines for proactive identification of trends which indicate degradation. This paper seeks to change the way the performance management is carried out. It identifies five best practices that have been classified as requirements to be included as part of software system design and construction to improve the overall quality of performance management of these systems in production. These practices were successfully implemented in a mission critical software system at design time resulting in effective and efficient performance management of this system from the time it was operationalized. 1 Introduction The business and technology landscape of today is characterized by the increasing presence of mission critical web applications. These web applications have progressed from being simple static content providing applications to application allowing all kinds of business transactions. The responsiveness of websites under concurrent load of a large number of users is an important performance indicator for the end users and the underlying business. The growing focus on high performance and resilience has necessitated including comprehensive performance management as an integral part of the software systems. However this is an area which has received limited focus and relies on a fix-it-later approach from a project execution perspective. The key to the successful performance management of critical systems in production is timely availability and accuracy of data which may then be analyzed for proactive identification of performance incidents. The inclusion of performance management requirements is essential in the design and construction phase of a software system. Our experience in handling of performance related incidents in critical web application over the last few years has shown that the focus on inclusion of such techniques starts when performance incidents get reported after the start of operations of the software system in production. This may be too late since there may be little room left for any significant design change at that point of time or may require a major rework in the application. This approach The copyright of this paper is owned by the author(s). The author(s) hereby grants Computer Measurement Group Inc a royalty free right to publish this paper in CMG India Annual Conference Proceedings. 19

2 is risk prone and expensive as making changes in implementation phase of software development lifecycle is difficult, incurs rework effort and a 100 times increase in cost [YOGE2009]. A number of studies have shown that responsive websites impact productivity, profits and brand image of an organization in a positive manner. In addition, slow websites result in loss of brand value due to negative publicity and decreased productivity. A survey done by Aberdeen Group on 160 organizations reported an impact of up to 16% in customer satisfaction and 7% in conversions i.e. loss of a potential customer due to a one second delay in response time. The survey also reported that the best in class organizations improved their average application response time by 273 percent [SIMI2008]. Satoshi Iwata et al. demonstrates the usage of statistical process control charts based on median statistic for detecting performance anomalies related to processing time in RuBiS which is a web based prototype of an auction site [SATO2010]. This performance anomaly detection technique requires the timely availability of measured value using appropriate instrumentation techniques (e.g. response time from the web application). This paper tries to bring about a paradigm shift from the prevalent reactive and silo-based approach in the domain of performance monitoring of mission critical software systems to an analytics based engineering approach by including certain proven requirements as part of the design and design process. The objective is to be able to know about a performance issue before a complaint is received from the end users. The silo-based approach analyzes several dimensions such as web applications, web servers, application servers, database servers, storage, servers and network components in isolation. The reactive approach involves including logs in a makeshift manner only when a performance incident occurs. This in turn results in the required information not being available at the right time for effective detection, root cause analysis and resolution of the problem. This paper suggests including practices like instrumentation, system performance archive, controlled monitoring, simulation and an integrated operational console as part of the design and design process of software systems. These requirements are not aimed at improving or optimizing the performance of the software system but are for enhancing the effectiveness and efficiency of performance monitoring in production. These requirements were successfully included as part of the design and design process in a mission critical e- government domain web application built using J2EE technology. This has helped the production support team to recognize early warning signs which may lead to a possible performance incident and take corrective actions quickly. The rest of this paper is organized as follows. Section 2 describes the application in which the proposed best practices were implemented. Section 3 describes the best practices in detail. Section 4 presents the results and findings of our work. Section 5 provides the summary, conclusion, limitations of our work and suggestions for future work. 2 Background These requirements were successfully included as part of the design and design process in a mission critical e- government domain web application built using J2EE technology servicing more than customers every day. The web application in this e-governance program is used by both external and department users for carrying out business transactions. The external users access this application on the Internet and the department users access it on the Intranet. The technical architecture has five tiers for the external users and three tiers for the department users. The presentation components are JSPs and the business logic components are POJOs developed using the Spring framework. External Users: The information flow from external users passes through five tiers. The tiers 1, 2A and 3 host the web application server. The tiers 2B and 4 host the database server. Tier 1 provides the presentation services. Tier 2A and 2B carry out the request routing from the tier 1 to tier 3 after carrying out the necessary authentication and authorization checks. The business logic is deployed in tier 3. Tier 4 holds all transactional and master data of the application. Department Users: The information flow from department users passes through three tiers. The tier 5 and 6 host the web application server. Tier 5 provides the presentation tier. The business logic id deployed in Tier 6. Tier 4 holds all transactional and master data of the application. 20

3 Figure 2 shows the logical architecture of the e-government domain web application. Figure 2: Logical architecture of the e-governance web application 3 Building Performance Management in Software Systems The existing approach of performance management of software systems in production involves a reactive and silo based approach. The silo based approach involves measurements at the IT infrastructure component level i.e. server, storage, web servers, application servers, database servers, network components and application server garbage collection health. The reactive approach involves adding log entries whenever a performance incident occurs. This section describes in detail five mandatory requirements that a software system needs to incorporate as part of the design and design process to ensure effective, proactive and holistic performance management after the system is in production. These requirements were successfully implemented as part of the design and design of a mission critical e-government domain web application with excellent results. This web application provides a number of critical business transactions to end users and requires meeting stringent performance and available service level agreement requirements. 3.1 Instrumentation The instrumentation principle of Software Performance Engineering states the need to instrument software systems when they are designed and constructed [CONN2002]. Instrumentation is the inclusion of log entries in components for generating data to be used for analysis. These log entries do not change the application behavior or state. Correct and sufficient instrumentation help in quick isolation of performance hotspots and determination of the components contributing to these hotspots. The logs from various tiers and sources form an important input to performance management of software systems in production. These logs can be from application or the infrastructure tier. The logs from the application tier include web server and custom logs of the web application. The logs from the infrastructure tier include processor utilization, disk utilization, application server garbage collector etc. The technique of implementing instrumentation includes use of filters, interceptors and base classes. Figure 3 shows the usage of a base class for implementing instrumentation. The software system requirements in the area of performance and scalability traditionally do not mention an instrumentation requirement. The experience of the authors in managing large scale software systems showed instrumentation being introduced as a reactive practice towards the end of the software development lifecycle for identification of performance incidents reported from the end users or performance tests. This reactive approach results in rework and schedule slippage due to the code changes need for instrumentation and regression testing 21

4 required following these changes. This paper recommends inclusion of this practice as a key requirement in the software requirements specification rather than being limited to being a best practice. PRACTICE: Include sufficient instrumentation in all tiers for quick isolation of performance problems and identification of the component(s) contributing to performance problems. public class TestBaseAction extends ActionSupport implements PrincipalAware, ServletRequestAware, SessionAware { public final String execute() throws Exception { String res; Date begin = new Date( ); res = execute2( ); Date end = new Date( ); logger.info(.end.gettime( ) begin.gettime( ).); } } Figure 3: A base class is a place to add instrumentation log entries. Figure 4 shows entries from a standard web server log. Each entry includes a timestamp, the request information, execution time, response size and status. The software requirements specification can explicitly state that the web server log need to be enabled for recording specific attributes. These entries can be aggregated for a time interval (e.g. two minutes) to arrive at statistics like count and mean response time or used for steady state analysis of the software system. In a stable system, the rate of arrival of requests is equal rate at which the requests leave the system. XXX XX - - [14/Aug/2014:22:06: ] "GET /OnlineApp/news/ticker.jsp HTTP/1.1" YYY XX - - [14/Aug/2014:22:06: ] "POST /OnlineApp/secure/AddressAction HTTP/1.1" YYY XX - - [14/Aug/2014:22:06: ] "GET /OnlineApp/images/bt_red.gif HTTP/1.1" XXX XX - - [14/Aug/2014:22:06: ] "GET /OnlineApp/images/bullet_gray.gif HTTP/1.1" XXX XX - - [14/Aug/2014:22:06: ] "GET /OnlineApp/status/trackingHTTP/1.1" ZZZ XX - - [14/Aug/2014:22:06: ] "GET /OnlineApp/css/doctextsizer.css HTTP/1.1" YYY XX - - [14/Aug/2014:22:06: ] "GET /OnlineApp/news/ticker.jsp HTTP/1.1" YYY XX - - [14/Aug/2014:22:06: ] "GET /OnlineApp/CaptchaRxs?x=1483d7f9s71990 HTTP/1.1" ZZZ XX - - [14/Aug/2014:22:06: ] "GET /OnlineApp/secure/ServiceNeeded HTTP/1.1" ZZZ XX - - [14/Aug/2014:22:06: ] "POST /OnlineApp/user/loginValidate HTTP/1.1" YYY XX - - [14/Aug/2014:22:06: ] "GET /OnlineApp/user/uservalidationHTTP/1.1" YYY XX - - [14/Aug/2014:22:06: ] "GET /OnlineApp/user/loginAction?request_locale=en&userName=AAA@GMAIL.COM HTTP/1.1" YYY XX - - [14/Aug/2014:22:06: ] "POST/OnlineApp/secure/logmeAction HTTP/1.1" Figure 4: Using Web Server Logs as an Instrumentation Tool Figure 5 shows entries from a custom web application log. Each entry includes an entry and exit timestamp, web container thread identifier, the request information, execution time, status and correlation identifier. The software requirements specification can explicitly state that the application web server log need to be enabled for recording specific attributes. The custom logs provide application specific information which may not correctly reflect in the web server log (e.g. a web server log may report an http status 200 but the business transaction may have 22

5 encountered a logical error). These entries can be aggregated for a time interval (e.g. two minutes) to arrive at statistics like count and mean response time or used for steady state analysis of the software system :01:13,836 WebContainer : DCBANKONL class.secure.action.uploadform - Mon Aug 14 21:01:00 GMT+05: ABBBA26d1ds :01:41,507 WebContainer : AAA@GMAIL.COM class online.secure.action.viewformaction - Mon Aug 14 21:01:34 GMT+05: s0ad0af :01:52,798 WebContainer : BBB@GMAIL.COM class online.secure.action.payment.paymentverificationaction - Mon Aug 14 21:01:46 GMT+05 : AAAA :02:34,466 WebContainer : CCC0990 class online.secure.action.creditcardpaymentaction - Mon Aug 14 21:02:26 GMT+05: AAAA :02:34,498 WebContainer : DDD@GMAIL.COM class online.secure.action.applicationsubmitaction - Mon Aug 14 21:02:19 GMT+ 05: aa000d02AAaA5a118s7 3.2 System Performance Archive Figure 5: Using Custom Logs as an Instrumentation Tool This practice involves keeping a record of the history of the software system by storing the values of various metrics related to the performance of the system. The metric provide a strong mechanism for reviewing past performance and identifying emerging trends. Brewster Kalhe founded the Internet archive for keep a record of the history of Internet [BREW1996]. The Http archive is a similar permanent record of information related to web page performance like page size, total requests per page etc. [HTPA2011]. The system performance archive must capture a minimum of three important attributes namely the measured value, the metric and the applicable domain for each measurement (e.g. the metric is the response time, applicable domain is the application home page and the measured value is 4.2 seconds). The measured metric can be explicit or implicit. An example of an explicit metric can be derived from the requirement that the software system shall be designed to process 99% of the online home page requests to complete within 5 seconds. This archive is used as input to in-house and third party analytical tools to carry out statistical analysis (e.g. mean, median, standard deviation, percentile) and modeling (e.g. capacity planning). This paper recommends inclusion of this practice as a key requirement in the software requirements specification. Critical software systems need to provision the required infrastructure for creating this archive in terms of compute and storage, in-house or third party analytical tools for this critical functionality at the time of design, capacity planning and construction. This compute and storage provisioning can be easily done on in a cloud environment. PRACTICE: Design and construct a system performance archive for critical software systems to keep a record of performance related information. 3.3 Controlled Monitoring This practice involves executing synthetic read only business transactions using a real browser, connection speed and latency. These synthetic business transactions can be executed from one or more regions. There are a number of incidents in which an end user reports experiencing slowness but the server health appears normal. The practice of controlled monitoring helps quickly determine if the incident is specific to the user reporting the problem. The software system requirements in the area of performance and scalability traditionally do not mention executing synthetic read only business transactions using real browser, at real connection speed and latency. This requirement can be implemented using frameworks like private instance of WebPageTest ( Critical software systems need to provision the required infrastructure for carrying out this monitoring and upload the results in the System Performance Archive. 23

6 PRACTICE: Use synthetic read only business transactions using a real browser, connection speed and latency to measure performance. 3.4 Simulation Environment This practice is based on the belief that most events happening in a system shall be reproducible under similar conditions. The causal analysis of certain incidents may remain inconclusive during initial analysis. Recreation of the symptoms leading to that incident, under similar conditions may lead to the deeper insight and help in finding the actual root cause. Since such simulation may not be feasible in the actual production environment in majority of the cases, a similar Simulation Test environment needs to be used. The prevalent practice in the Industry appears to be to treat performance test as single or one time activity prior to implementation resulting in provisioning of a simulation environment only for limited duration. As a result, reproduction of a complex problem that occurred in production becomes extremely difficult. PRACTICE: Provide a simulation environment to reproduce the performance incident in production like conditions to ensure completeness and correctness of the causal analysis of that incident. 3.5 Integrated Operations Console In order to manage performance of a production system effectively, it is essential that production support teams have the ability to visualize anomalies and resolve exceptions without delay. The prevalent silo based approach involves measurements at the IT infrastructure component level i.e. server, storage, web servers, application servers, database servers, network components and application server garbage collection health. The concept of an Integrated Operations Console can be very effective in such scenarios. This console not only monitors the system performance, but also records exception conditions and provides ability to take actions to resolve these conditions. The typical actions may range from killing a process or query to restarting of a service. This console shall also need to provide a component level checklist which can be executed automatically prior to start of operations every day. Table 1 shows an extract of a database checklist. This console empowers the teams to take quicker action once an exception is observed. Besides, allowing actions, the console may automatically gather the relevant data such as heap dumps, database snapshot to aid further investigation. Host accessible? Instance available? Able to connect to the database?.... Table 1: Using Custom Logs as an Instrumentation Tool PRACTICE: Provide an Integrated Operations Console for monitoring the system performance parameters and mechanism to resolve anomalies for which resolution processes are known. 24

7 4 Results & Findings The above five practices were successfully implemented as part of the design and design of a mission critical e- government domain web application. 4.1 Implementation 1 The instrumentation was implemented as an integral part of the design and design activity of the e-governance application. Figure 6 shows the instrumentation implemented in that application. The tiers 2, 2A, 3, 5 and 6 implemented instrumentation in the form of web server logs and custom logs. The tiers 2B and 4 implemented instrumentation in the form of database snapshots. The relevant logs from all tiers all collected at a shared location. The information from these logs are processed and used as input to the System Performance Archive and Integration Operations Console. Figure 6: Instrumentation implemented in web server, application server and database The first example shows how simple instrumentation helped finding the cause of a performance incident in where end users experienced high response time. Figure 7 and Figure 8 show request and response time graphs of tier 1 and 3 respectively calculated using the web server logs and depicted on the Integrated Operations Console. The request is the count of all the requests serviced in a given time interval and the response time is the mean execution time of all the requests serviced in that time interval. The visual inspection of Figure 7 clearly shows a high mean response time in tier 1. The spike in the mean response time is not visible in tier 3 for the same duration, but there is drop in the number of request serviced by this tier. This helps us to conclude that the origin of the performance incident may be tier 1 or 2A. 25

8 Figure 7: The request and response time graph of tier 1 from the Integrated Operations Console. The request is the count of all requests serviced in a given time interval and the response time is the mean execution time of all requests serviced in that time interval. Figure 8: The request and response time graph of tier 3 from the Integrated Operations Console. The request is the count of all requests serviced in a given time interval and the response time is the mean execution time of all requests serviced in that time interval. The second example shows how the same instrumentation helps in determining whether the system is in a steady state. Figure 9 and Figure 10 shows that the system is in a stable state or equilibrium as the number of arrivals is equal to the number of exits. The arrivals and exits graphs are depicted on the Integrated Operations Console. Figure 9: The count of arrivals in a given time interval from the Integrated Operations Console 26

9 4.2 Implementation 2 Figure 10: The count of exists in a given time interval from the Integrated Operations Console The provisioning of a simulation environment even after implementation helped resolve a serious long stop the world garbage collection pauses in the application server on tier 1 and 3. The simulation and confirmation of resolution of the issue required multiple cycles of test execution. The identification of the reason for these pauses as a class unload problem required adding debug logs in a Java runtime class. Executing multiple test cycles to reproduce the problem is not feasible in the development or production environment. Figure 11 shows the garbage collector log show more than classes getting unloaded. Figure 11: The garbage collection log from the Simulation Environment 27

10 4.3 Implementation 3 The System Performance Archives provides an insight into the historical as well as emerging performance related trends for the software system. These trends are crucial to assess the capability of the software system to render services while meeting the required performance objectives. This information is also used in capacity planning exercises. Figure 12 shows the implementation of System Performance Archives in the e-government domain application. Figure 12: Implementation of the System Performance Archive Certain trends may be cyclic and may appear only after a particular period. Other trends may be more permanent in nature and tend to grow or decline. The implementation used descriptive statistics like count, mean, median, minimum, maximum, percentile and standard deviation. Figure 13 shows the mean response time trend of a business transaction (BT1) for a period of a month. The range of this response time is between 800 to 900 milliseconds. As can be clearly seen, the response time remained constant for the complete month. Figure 14 shows the mean response time trend of a second business transaction (BT2) for the same month. It can be seen that there is a change in the trend from 7 th (1919 milliseconds to 2004 milliseconds) and on 29 th (2176 milliseconds to 2813 milliseconds). The changes in the trends were further investigated to find out the reason for the same. The change in the trend on 7 th was due to additional business logic being added to the business transaction as part of the deployment on 6 th. The increase in the mean response time on 29 th was due to a network issue resulting in transactions getting executed slowly. Figure 13: Daily mean response time trend of a business transaction (BT1) for a month 28

11 5 Threats to Validity Figure 14: Daily mean response time trend of a business transaction (BT2) for a month The practices described in this paper are based on the authors experience in working and managing mission critical web application. These practices may need to be augmented with additional practices which the software performance engineering community may share. 6 Conclusions In this paper we have introduced a few mandatory requirements required to be included as part of the design and construction of a mission critical software system. This will bring about a paradigm shift in the prevalent practice of production support teams being equipped adequately to quickly detect a performance incident, gather enough information during an incident or reproduce the incident for a more accurate closure. These practices when included as part of design provided significant benefits in production support of a mission critical e-government domain system in areas of timely detection of a performance incident allowing corrective action, visualizing emerging trends and providing a more correct closure to incidents. 7 Future Work The future work in this area includes creating baselines and statistical models for metrics like response time and throughput. There is also a need to devise proactive anomaly detection models using techniques like steady state analysis. 29

12 References [YOGE2009] K. K. Aggarwal and Yogesh Singh, Software Engineering, New Age International Publishers, p470, [SIMI2008] B. Simic, "The Performance of Web Applications: Customers Are Won or Lost in One Second", Technical Report - Aberdeen Group, Accessed on 31 Jan 2014 at [SATO2010] S. Iwata and K. Kono, Narrowing Down Possible Causes of Performance Anomaly in Web Applications, European Dependable Computing Conference, p , [CONN2002] Connie U. Smith, Performance Solutions A practical guide to creating responsive, scalable software, Addison Wesley, p243, [BREW1996] Brewster Kahle, Internet Archives, Accessed on 17 Aug 2014 at [HTPA2011] http archive, Accessed on 31 Jan 2014 at [WPGT] WebPageTest, Accessed on 31 Oct 2014 at 30