WHITEPAPER A Quick guide to understanding & evaluating APM Tools
Table of Contents APM An Overview 3 How APM works? 3 Key features of APM 4 Monitoring Profiling 4 5 Others 6 Evaluation Matrix with Scores 7 Evaluation Matrix 7 Classification of Scale for Scores 8 9 APM tools: Features, Limitations & More Practical limitations/ drawbacks of using APM Tool Increases the network traffic 11 11 References 11 12 Author information 12 The Final Word Author information Gunavant Jambu is a Technical Architect with Hexaware Technologies. The paper has been internally reviewed by Sanjay Sharma, Head of Consulting and Solutions (ADM team).
APM An Overview Application Performance Management primarily refers to management and monitoring of the availability and performance of applications. Most frequently, when users experience performance issues, it could be due to issues with network, server, database, an application or a code. It is very difficult to identify the root cause of the issue. This is where APM can give you an advantage. The APM tool provides an ability of the baseline experience and notifies the infrastructure team, whenever the system moves out of response time. This provides IT an ability to quickly troubleshoot what the issue is or where it is coming from and stops its impact on the system. This document covers the detailed features of various APM tools and also takes you through a comparative evaluation matrix of these tools. How APM works? The APM tool has various kinds of agents such as application agent, server agent, database agent and end-user agent. Every agent has its own functionalities. Application agent will send data for applications such as Java,.Net, PHP, Python, etc. The DB agent will send monitoring information related to database. The Server agent will send data related to CPU, Memory, IO, Network IO, etc. The End-user agent will send data related to Web-user, mobile-use, load-time, etc. We need to set up and configure these agents to the server. Once the server starts, the agent will start sending data to the APM Cloud server. It will store the data in the database and the user can view the reports using the APM Web-module. Process time can be measured in two ways - Figure1 : High-level Flow Diagram MONITORING The APM tools continuously discover and monitor all the processing in an application environment. They provide information related to application traffic and also help users to identify bottlenecks.
Application Performance Monitoring The APM tools monitor the health of an application. Different tools have different algorithms to monitor the health. For example, New Relic uses Apdex code algorithm. Apdex algorithm finds application/server response time by using a pre-configured threshold value and the threshold value for AppDynamics is updated, based on the live traffic of the servers. Back-end Monitoring The APM tools provide back-end monitoring provisions. Back-end means database and remote services such as message queues. Some of the APM tools automatically find out the back-end components and start monitoring & profiling. It helps to troubleshoot slow database and remote service calls as well as expensive SQL Statements and Database connection problems. Background Task Monitoring Background tasks are like cron jobs. This kind of a job executes and runs at a specific time or interval without any user intervention. Some of the APM tools automatically identify the background jobs and start monitoring & profiling. Infrastructure Monitoring The APM tools monitor hardware and servers. The tools monitor CPU usage, memory utilization, disk read & write and network IOPS. They also monitor applications and Web servers such as IIS, Tomcat and JBoss. End-User Monitoring End-user monitoring provides information of the end-user s experience. The APM tools provide monitoring for the Web browser as well as the mobile application. It helps to identify region-wise the heaviest load and the slowest end-user response time. User performance can vary depending upon the Web browser and the mobile being used. Alert and Respond This is a very good feature of the APM tool. Alert will occur when a system is running under threshold value or the performance is below average. In such cases, the APM tool will send a mail notification to the user. Responses help you automate preventative actions to resolve the issues before they slowdown a server or cause outage. PROFILING The APM tools also offer profiling, which is a detailed level of monitoring and takes place before an application is deployed live. This APM tool provides two kinds of profiling approaches - 1) Offline 2) Real-time. Some of the tools only provide real time approach, while some provide both approaches. We can use the offline profiling during the development phase and the real-time approach whenever any performance issues occur in the production environment. In the real-time approach, an agent is installed at the server side, which continuously sends data to a server and the server analyzes the present data and converts it into rich UI. Process Time Process time can be measured in two ways - Wall-clock time: Wall-clock time means the time difference between the entry and the exit of the method. CPU Time: The real time that was spent in the method by the CPU is called the CPU time. The wall-clock time is not the time the CPU has actually spent in that method because the CPU handles many threads with different priorities. The scheduler of the OS (Operating System) can interrupt the execution of a method and perform other tasks. In extreme cases, the CPU time and the wall-clock time can differ by a large factor. Most of the profiling tools like JProfiler provide both profiling mechanisms. Process time is used to identify which method or process is taking how much time. It provides a call tree method, which makes it easy to identify the process time. By using this, the user can identify the methods which are taking more CPU time.
The wall-clock time is not the time the CPU has actually spent in that method because the CPU handles many threads with different priorities. The scheduler of the OS (Operating System) can interrupt the execution of a method and perform other tasks. In extreme cases, the CPU time and the wall-clock time can differ by a large factor. Most of the profiling tools like JProfiler provide both profiling mechanisms. Process time is used to identify which method or process is taking how much time. It provides a call tree method, which makes it easy to identify the process time. By using this, the user can identify the methods which are taking more CPU time. Memory Utilization The APM tools keep the records of the created object. They help to identify which object is costly. Memory profiling helps to provide details on the root cause of the memory leakage. Most of the APM tools provide memory profiling for Java language only. Thread State This APM tool displays the current state of a thread. In multiple threaded environment, there could be chances of a deadlock. The APM tool can find out the deadlock thread that caused the deadlock. GC Activities The APM tools also provide GC activities. GC stands for Garbage Collector. This is a costly process, which is performed by JVM internally. This process is used to remove all the objects which are not required. This APM tool provides details of the GC, such as total time and process time of the GC. This helps to find out memory leakage after the GC activity. OTHERS 1. Licenses APM tools can have trial/free/paid licenses. The tools provide the software for testing for some trial period (with some limited features). Once the trial period is over, the product is to be bought. Different tools have different payment models. For example with New Relic, Pro account pricing starts at $199/month per host ($149 if opting for a yearly plan). This includes APM, Servers, Platform and Browser basics. Mobile monitoring costs $49/ month ($29 if opting for a yearly plan) with 1 week of data retention. The Insights product starts at $250/month for up to 75 million events. ricing with AppDynamics Pro programs is individual-based and you will have to contact sales to get a customized plan, depending upon the number of agents you require. Mobile monitoring is a bit different and each agent is priced per 5,000 active users in a month. Sematext is 4-5 times less expensive [REF 4]. You need to contact the support team for detailed pricing. JProbe is an apachefree license, which is a free and open source software licensing agreement from the Apache Software Foundation. It does not include any cost to download and uses JProbe Tool. JProfiler has two types of licenses - 1) Single license and 2) Floating license. Single license is for one machine. We cannot install it on another machine. On the other hand, the Floating license can be installed on multiple machines. For details, please refer [REF. 5] 2. Integration Mostly there are two kinds of APM Integration Models - 1. SaaS-based 2. On-premise. 1) SaaS-based (Software-as-a-Service): The software is deployed to the Web and we can monitor the server by using the Web application. 2) On-premise: Software is installed on the user s premises.
3. Platform Support The APM tool can also be installed as a mobile app. Some of the APM tools provide mobile capabilities. Please refer to the matrix ahead for more details. 4. User Interface Rich User Interface is the key component of any tool. Presentable user interface makes the tools unique. It also depends on the end-users test. 5. Dashboard and Reports Dashboard and Reports provide different kinds of charts/ layout to analyze data. End-user can also drill down into the report further to view it in detail. 6. Supported Language Selection of APM also depends on the application language. Some of the APM tools support profiling on all the application languages like Java,.Net etc. and some support only a specific language. Please go through the Evaluation Matrix below for quick comparison between the various APM tools. Evaluation Matrix with Scores Following matrix shows scores against key evaluation features published by the respective vendors: Evaluation Matrix Features AppDynamics Ruxit Sematext Jprobe Jprofiler New relic License Trial 5 5 5 0 5 5 Free 0 0 0 5 0 0 Paid 4 5 5 0 4 4 Integration SaaS 5 5 5 0 0 5 On Premises 5 0 5 5 5 0 Platform Support Web-based 5 5 5 0 0 5 iphone App 0 5 0 0 0 5 Android App 0 5 0 0 0 5 Window Phone App 0 0 0 0 0 0 Window Desktop 0 0 0 5 5 0 Monitoring Application Performance Monitoring 5 5 5 0 0 4 Back-end Monitoring 5 5 5 0 0 4 Background Task Monitoring 5 5 5 0 4 3 End-User Monitoring 5 5 4 0 0 4 Infrastructure Monitoring 5 5 5 0 0 5 Profiling Process Time 4 0 2 5 5 4 Memory Utilization 4 0 2 5 5 3 Thread State/ Deadlock 4 0 0 5 5 0 GC Activities 0 0 0 5 5 0 Troubleshoots Errors 5 0 0 5 5 5 Alert and Respond 5 5 5 0 0 5
Features AppDynamics Ruxit Sematext Jprobe Jprofiler User Interface 5 5 4 3 5 4 Dashboard and Reports 5 0 1 0 0 5 Supported Language Java 5 5 5 5 5 5 C++ 5 0 0 0 0 0.Net 5 5 0 0 0 5 PHP 5 5 0 0 0 5 Node JS 5 5 5 0 0 5 Python 5 0 0 0 0 5 Ruby 0 5 0 0 0 5 Database 5 5 5 3 3 5 Web Server 5 5 5 0 0 5 Cloud Support APM Matrix by Category New relic Figure 2: JProfiler has the highest score in the profiling category and AppDynamics has the highest score in the monitoring category. Classification of Scale for Scores Criteria Score If the considered feature is not available 0 If the considered feature is available in the tool and its feature implementation is only partial 1 If the considered feature is available in the tool and its feature implementation meets key critical use cases 2 If the considered feature is available in the tool and its feature implementation meets most of the 3 use cases If the considered feature is available in the tool and its feature implementation all of the use cases and also adds additional features 4 If the considered feature is available in the tool and its feature implementation all of the use cases and also adds additional features with good quality and performance 5
APM tools: Features, Limitations & More Mostly all the tools serve the basic monitoring features, but there are some key features, which can help to decide on which tool to select. 1. AppDynamics a. Features i. Supports Linux, Mac, Windows ii. Has SaaS and on-premises model iii. Has its own solution to monitor application performance. It automatically creates a dynamic baseline for apps performance, which varies by time. iv. Deep code level diagnosis using Takipi makes AppDynamics different from the other tools v. Can profile an application to find out memory leak issues vi. Has very nice and rich user interface, which makes it easy to understand and easy to drill down further into the reports b. Limitation i. Profiling is not as detailed as compared to JProfiler ii. Requires TAPI (third-party tool) for deep code level diagnosis iii. Does not support Android, Windows and mobile app platforms 2. New Relic a. Features i. New Relic is using the Apdex score index algorithm. This algorithm uses T (a pre-defined threshold for response time) to find out end-user satisfaction. Here s an example of the way this score is calculated: The Apdex score is a ratio value of the number of satisfied and tolerating requests to the total requests that were made. Each satisfied request is counted as one request and each tolerating request counts as half a satisfied request. For Example: a) During a 1 minute period, a server handles 100 requests. b) The threshold T = 500ms. This value is configurable by user. c) 50 of the requests were handled within 500ms, so they are classified as Satisfied. d) 30 of the requests were handled between 500ms and 2000 ms, so they are classified as Tolerating. e) The remaining 20 were not handled properly or took longer than 2 seconds. Thus, they have been classified as Frustrated. f) The resulting Apdex score is 0.65: (50 + (30/2))/100 = 0.65 ii. Supports iphone, Android and Web-based platforms iii. Supports monitoring for Ruby, PHP, ios,.net, Java, Android Apps, Python and Node.js iv. Has SaaS-based integration v. Provides rich user interface to display performance of the server & applications vi. Provides profiling to find out the root cause of the memory and application performance issues vii. Can integrate Bitium, Datalog, Jira, OpsGenie, Slack & VictorOps. b. Limitation i. New Relic is costlier than Sematext ii. Apdex is a fatally flawed mechanism. It uses static threshold to monitor the performance. Performance of the application can vary depending upon scenarios such as load, geography, environment, etc. iii. Detail-level application and memory-level profiling are not supported iv. Requires third-party tool (TAPI) to view source code v. New Relic has only a SaaS model. It does not have on-premise integration.
Practical limitations/ drawbacks of using APM Tool Increases the network traffic APM agents continuously send server, CPU and memory data to the server, leading to increased network traffic. THE FINAL WORD There are mainly two categories of the features - Monitoring and Profiling. Tools like AppDynamics, New Relic, Sematext, Ruxit are very good in monitoring but average in profiling; while tools such as JProfiler and JProbe are very good in profiling but not preferable for monitoring. The purposes of using monitoring and profiling tools are different. Profiling tools are mainly used at the time of development to improve the CPU performance and to minimize the memory leak issues, while monitoring tools are primarily used to monitor response time of the server, hardware and database components. So, depending upon our requirements, we need to make a choice between the profiling and the monitoring tool. Currently, all the monitoring tools are evolving their features. Core feature-wise, all the monitoring tools are approximately on the same benchmarks. Apart from the core features, monitoring algorithm, UI, pricing model, integration and application language support are the key factors to consider while selecting a monitoring tool. In the profiling tools, JProfiler is the most advanced profiling tool but it s not free. On the other hand, JProbe is a free software with basic features but the user interface is not as good as JProfiler. Hence, it is recommended to use AppDynamics for monitoring and JProfiler for profiling. References https://docs.appdynamics.com/display/pro14s http://resources.ej-technologies.com/jprofiler/help/doc/ https://dzone.com/articles/appdynamics-vs-new-relic-%e2%80%93 https://sematext.com/img/spm-vs-newrelic-comparison.pdf https://www.ej-technologies.com/buy/jprofiler/select https://www.ej-technologies.com/products/jprofiler/docs http://www.javaperformancetuning.com/tools/jprobe/ https://sematext.com/spm/ https://newrelic.com/application-monitoring http://www.dynatrace.com/en/ruxit/capabilities/
About Hexaware Hexaware is one of the leading, global providers of IT, Application, Infrastructure, BPO and Digital services. Our business philosophy of Shrink IT, Grow Digital allows customers to significantly shrink commodity IT spend while partnering with them to embrace digitalization. The Company focuses on key domains such as Banking, Financial Services, Capital Market, Healthcare, Insurance, Manufacturing, Retail, Education, Telecom, Travel, Transportation and Logistics. Hexaware focuses on delivering business results and leveraging technology solutions by specializing in services like; Application support, development and maintenance, Enterprise Solutions, Human Capital Management, Business Intelligence & Analytics, Digital Assurance (Testing), Infrastructure Management Services, Digital and Business Process Services. Founded in 1990, Hexaware has a well-established global delivery model armed with proprietary tools and methodologies, skilled human capital and SEI CMMI-Level 5 certification. For additional information logon to: www.hexaware.com NA Headquarters Metro 101, Suite 600,101 Wood Avenue South, Iselin, New Jersey - 08830 Tel: +001-609-409-6950 Fax: +001-609-409-6910 India Headquarters 152, Sector 3 Millennium Business Park A Block, TTC Industrial Area Mahape, Navi Mumbai 400 710 Tel : +91-22-67919595 Fax : +91-22-67919500 EU Headquarters Level 19, 40 Bank Street, Canary Wharf, London - E14 5NR Tel: +44-020-77154100 Fax: +44-020-77154101 APAC Headquarters 180 Cecil Street, #11-02, Bangkok Bank Building, Singapore 069546 Tel : +65-63253020 Fax : +65-6222728 Safe Harbor Statement Certain statements in this press release concerning our future growth prospects are forward-looking statements, which involve a number of risks, and uncertainties that could cause actual results to differ materially from those in such forward-looking statements. The risks and uncertainties relating to these statements include, but are not limited to, risks and uncertainties regarding fluctuations in earnings, our ability to manage growth, intense competition in IT services including those factors which may affect our cost advantage, wage increases in India, our ability to attract and retain highly skilled professionals, time and cost overruns on fixed-price, fixed-time frame contracts, client concentration, restrictions on immigration, our ability to manage our international operations, reduced demand for technology in our key focus areas, disruptions in telecommunication networks, our ability to successfully complete and integrate potential acquisitions, liability for damages on our service contracts, the success of the companies in which Hexaware has made strategic investments, withdrawal of governmental fiscal incentives, political instability, legal restrictions on raising capital or acquiring companies outside India, and unauthorized use of our intellectual property and general economic conditions affecting our industry. www.hexaware.com marketing@hexaware.com