Real Time Monitor of Grid job executions

Transcription

1 Journal of Physics: Conference Series Real Time Monitor of Grid job executions To cite this article: D J Colling et al 2010 J. Phys.: Conf. Ser View the article online for updates and enhancements. Related content - CMS Dashboard Task Monitoring: A usercentric monitoring view Edward Karavakis, Julia Andreeva, Akram Khan et al. - On glite WMS/LB monitoring and Management through WMSMonitor D Cesini, D Dongiovanni, E Fattibene et al. - A Grid job monitoring system Catalin Dumitrescu, Andreas Nowack, Sanjay Padhi et al. Recent citations - Cloud Monitoring : Peer Hasselmeyer et al - A Self-adaptive hierarchical monitoring mechanism for Clouds Gregory Katsaros et al This content was downloaded from IP address on 30/11/2017 at 08:29

2 Real Time Monitor of Grid job executions D J Colling 1, J Martyniak 1, A S McGough 1, A Křenek 2, J Sitera 2, M Mulač 2 and F Dvořák 2 1 Imperial College London, Prince Consort Road, London SW7 2BW, UK 2 CESNET, Zikova 4, Prague 6, Czech Republic. d.colling@imperial.ac.uk, janusz.martyniak@imperial.ac.uk, asm@doc.ic.ac.uk, ljocha@ics.muni.cz, sitera@civ.zcu.cz, valtri@civ.zcu.cz Abstract. In this paper we describe the architecture and operation of the Real Time Monitor (RTM), developed by the Grid team in the HEP group at Imperial College London. This is arguably the most popular dissemination tool within the EGEE [1] Grid. Having been used, on many occasions including GridFest and LHC inauguration events held at CERN in October The RTM gathers information from EGEE sites hosting Logging and Bookkeeping (LB) services. Information is cached locally at a dedicated server at Imperial College London and made available for clients to use in near real time. The system consists of three main components: the RTM server, enquirer and an apache Web Server which is queried by clients. The RTM server queries the LB servers at fixed time intervals, collecting job related information and storing this in a local database. Job related data includes not only job state (i.e. Scheduled, Waiting, Running or Done) along with timing information but also other attributes such as Virtual Organization and Computing Element (CE) queue - if known. The job data stored in the RTM database is read by the enquirer every minute and converted to an XML format which is stored on a Web Server. This decouples the RTM server database from the client removing the bottleneck problem caused by many clients simultaneously accessing the database. This information can be visualized through either a 2D or 3D Java based client with live job data either being overlaid on to a 2 dimensional map of the world or rendered in 3 dimensions over a globe map using OpenGL. 1. Introduction The Grid provides a mechanism for large computational tasks to be performed in a distributed manner over physical and geographical barriers. This allows individuals, who use the Grid to be abstracted away from the requirements to locate, provision and deploy their work to large scale, distributed, computing resources. These Grid users can then take advantage of greater computational power than they themselves would be able to own or locally have access to. However, due to the remote nature of the Grid it is difficult to visualize what the Grid is, its current state or how active it is. The role of the RTM is to address these issues. It provides a useful dissemination tool which allows non-technical people to quickly appreciate the scale of the Grid but also allows for the tunneling down on specific sites to provide useful information on the state of these sites and the status of jobs running on these sites. Individual data on specific jobs is captured which can be later analyzed in order to determine the performance of the job. c 2010 IOP Publishing Ltd 1

3 The Real Time Monitor is, to date, the only tool that has a privileged access to distributed Logging and Bookkeeping servers of the EGEE Grid. It makes it a very useful tool not only for individual users but also for experimental communities. It provides an important source of information for CERN Dashboard [2], the system which provides the Large Hadron Collider (LHC) experiments a single entry point to monitoring data from distributed computing resources used by LHC Virtual Organizations (VOs). 2. Architecture The Real Time Monitor architecture is presented in figure 1. The RTM database is the central element of the system. It contains not only current state of Grid jobs monitored by the system but also holds information about various EGEE Grid infrastructure elements (services) and their geographical position. The Grid services of interest are Resource Brokers (RB), Workload Management Systems (WMS), Computing Elements (CE) and Logging and Bookkeeping (LB) Servers. These are grouped into sites. Information about a site and its components is based solely on data available from Berkeley Database Information Index (BDII) [3] services and GOC database [4]. The RTM component responsible for keeping site information up to date is the BDII Scanner. It runs periodically 4 times a day performing LDAP search requests to BDII servers. We support multiple BDIIs thus increasing the worldwide coverage of monitored sites. There are certain requirements for a site to be included into the monitoring system. Firstly it has to be constantly present in a BDII (we allow for short breaks but sites which are not present in a BDII for longer than 3 days are disabled). Secondly it has to publish a correct geographical location, so that it can be positioned on a map. New sites once approved by the RTM management team appear on a map automatically. A site may publish several RB/WMS and CE nodes. Those are represented on a map by geographically located rotating triangles and flashing circles respectively. LB servers Server RTM Database (PostgreSQL) Summary files site rb/wms ce jobs Enquirer Apache Server rrdtool (graphs) html/xml BDII Scan BDII Figure 1. Real Time Monitor architecture diagram The LB servers on the other hand, although they play a key role in the RTM system are not presented on a map. They are used to obtain near to real time information about state of jobs on the Grid. This information is gathered by a multithreaded RTM Server component. The server runs on a dedicated computer at Imperial College London and has privileged access to the worldwide LB servers. The RTM Server component queries all monitored LB servers once every minute and feeds jobs information into the database. We record the unique job identifier, the RB/WMS server that handles the job, it's LB server, VO, the CE, job queue name on the CE and the job registration time. In the early job lifetime period some of the properties described above might not be known. A single job is represented in the database by one record, which is updated with new state information when a state change is observed on the LB responsible for that job. We keep the time when the state change occurs. 2

4 When jobs terminate (successfully or not) a job summary record is downloaded from the LB and written out by the Server. It contains all the states the job passed through with their associated timestamps. Since this includes more details than the database job records it is better suited for off line data analysis of Grid job executions. An example of such analysis is presented below, and an approach to use RTM data to studying EGEE production grid workload is presented here [5]. In order to make the RTM data stored in the database available to potentially many clients the Enquirer components has been developed. It reads jobs information from the database and stores it in XML format in an Apache Web Server area. Clients can contact the Web Server to download and process this data. It separates the RTM database from client requests, and thus reduces the load on the database. The Enquirer also provides information used to produce daily CE reports which are published on the RTM webpage [6]. A detailed CE report is also produced every 5 minutes and may be displayed on request. This topic will be covered in greater detailed in the section describing RTM Java client display programs. In order to examine potential new sites and LB server candidates to be monitored by the RTM a dedicated LB checker program runs once per day and informs the RTM management team about newly visible LB servers which have not opened their databases for the RTM to use. The LB administrators are ed and offered a possibility to add their site to constantly growing number of monitored GRID resources. 3. Real Time Monitor Client Programs 3.1. RTM monitored resources on the Web. The Grid resources available to the Real Time Monitor are published on the Web [7] by the BDII Scanner. This webpage is a formatted as an expandable list of RB/WMS and CE nodes ordered alphabetically by country and site. By following a CE or WMS link one can obtain detailed job information for all jobs currently present in the database. As mentioned in the previous section this list is automatically synchronised with resource description available from BDII systems. Figure 2. View of the RTM information in 3 dimensions. 3

5 3.2. Java client programs. The RTM system supports 2 visualisation client programs. The first superimposes jobs activity over a 2 dimensional map of the world. It is written entirely in Java, which ensures high level of portability. It can be run as a standalone application or as a Web applet. The second offering is a Java based program that is capable of displaying Grid jobs rendered over a 3 dimensional globe with satellite maps of the Earth to give a locational context. This second program uses OpenGL, which reduces machine independence. This tends to require the use of a computer with a higher end graphics cards. However, both client programs are able to display job activity with a similar level of accuracy. These Java clients, which we make available for download, utilise the same information sources. Using satellite imagery from NASA (figure 2). These programs display the Grid as it is geographically distributed over the world. In near real time (there is a 3 minute delay due to gathering and processing of the data) they show running and scheduled jobs. Sites are represented by pulsating circles, which are proportional in size to the number jobs at the site. The colours used signify the state of the jobs, green is used to show running and magenta represents scheduled jobs. We also show jobs transfers between WMS/RB nodes and CEs. These are shown as lines with moving arrows indicating a direction of job flow from or to a WMS/RB. The outgoing, scheduled job traces are drawn in magenta, while yellow or red colour is used for jobs that are returned to the RB/WMS by a CE. A yellow line indicates a successful job while a red one denotes job failure. Both programs allow moving around the map of the Earth, zooming in on a location (figure 3), and isolating a particular VO or RB/WMS. More detailed information about jobs running at a site may be obtained by clicking on the site (figure 4). A sites information window is displayed with site name and logo. There are several buttons, one for each RB/WMS or CE, when clicked a new window with a sort-able table of jobs attributes appears. The lower part of the site info window contains a VO colour coded histogram of jobs running on site s CEs in a 6 hour time window. Some sites which joined the Grid Guide project [8] have an additional button that enables you to see the Grid Guide webpage for that site in an embedded browser. Users can explore an interactive map of the world and visit a sample of hundreds of institutes using Grid computing to power their research. Figure 3. Zoomed-in 3D view 4

6 Figure 4. Site information display showing jobs on CEs separate for each VO 4. Analysing data collected by the RTM During our work on RTM we have been able to collect over 70 million job executions since The Job summary information is collected and recorded by the Server (figure 1). By post-processing we are able to determine the life-cycle of each job submitted through EGEE Grid. Note that this information is not complete due to data protection issues some information must be anonymised and information can t be retrieved form sites for which the RTM does not have access. We present here analysis of job life-cycles gathered between September 2005 and October These are derived from summary files of some 30 sites in 2005 scaling up to over 300 sites by late As an example we will present results from the LHCb VO to illustrate key characteristics. We define here a number of characteristics which we use in analysing the job life cycles. All job life-cycles were post-processed into ROOT [9] tables for this work: E USER : is the users impression of the efficiency of a job and is defined as the job execution time divided by the time it is in the system. E RAW : is the proportion of jobs which execute successfully and is defined as the total number of successful jobs in time T divided by the total number of jobs in time T. Figure 5 shows a stacked histogram of total number of successful jobs (in green) and failed jobs (in red) in the top and raw job efficiency at the bottom. LHCb use pilot jobs, which first try to determine if the resource is appropriate before trying to run a real job. If they find the resource unsuitable or there is no real job to run they fail quickly. This generates a large number of failed jobs in the top histogram of the figure and a low raw efficiency in the bottom histogram. 5

7 Figure 5. Job Submissions vs. time and E RAW vs. time for LHCb VO When we look at the time spent on job execution the picture is quite different (figure 6). Since pilot jobs use virtually no time to run the time used by these failed jobs is small. As a consequence the user efficiency is close to 1. Figure 6. Job Hours vs. time and E USER vs. time for LHCb VO 6

8 5. Recent development As described above, LB [10] is a key service supplying job data to RTM. However, the current implementation of RTM access to LB data evolved gradually for several years, and it now exhibits several known problems: - An architecture of one thread per LB does not scale well. There are dozens of active LB's in EGEE, and memory requirements and thread management overhead of the LB access module has become critical. - The module accesses the LB internal database directly. However, it is not a public interface and yields inaccurate results. With the next major LB release (2.0) the use of this interface will no longer work due to changes in the database schema. - Database queries are suboptimal, possibly generating unsustainable load on LB machine, and blocking WMS/LB operation. - Granting RTM privileged access to LB databases is an unnecessary escalation of privileges the table-level privileges grants the RTM access to much more information than it requires. For these reasons we decided to replace the LB access module with a new component LB harvester', implemented from scratch, and also re-used by other high-level monitoring tools (such as the Experiment Dashboard, [11]). This module uses public LB interfaces only, and it aims at maximal efficiency. LB makes the job data available in two modes: - Query (pull): the user specifies conditions (e.g. all my jobs submitted to a specific CE last Friday), server returns the set of matching jobs. - Notification (push): the user subscribes with conditions and listener endpoint; when any job enters a state matching the specified conditions, a message is sent to the user's listener. A persistent message delivery mechanism also deals with temporary unavailability of the listener. Because repeated queries, which would be necessary for RTM operation, return significantly large results for each query even thought there is usually only a small amount of new information, we are focused on the notifications. However, as the system was not originally designed for this usage pattern (virtually any job state change, i.e. fairly large amount of data is sent over long lived channel) several extensions and optimisations were required on LB side. A LB harvester instance reads the RTM database to retrieve the list of active LBs (figure 7). Then it registers with them to receive notifications, and it maintains the active registrations. Due to the persistent notification delivery restart of the harvester or crashes of the machine where it runs do not result in loss of data. On the other hand, if the harvester is not restarted within the lifetime of notification registration (several hours), the messages are dropped by LB server. In this case the harvester, on its restart, catches up with a LB query going back in time until the timestamp of the last received message. The incoming messages are processed and stored onto the internal RTM database. Data is accessed from there by other unchanged RTM components. The architecture is multi-threaded to leverage multi-core machines. On the other hand, a single thread may serve multiple connections to avoid unnecessary overhead. In this way all the previous issues are addressed: - One harvester thread accepts messages from multiple LB's. A limiting factor is the total number of incoming messages, not the number of connections. - A message on a specific job is generated only on job state change. There are no more massive queries to LB. - Access to LB uses its stable public interface only, not dependant on changes in LB internals. Our stress tests of the transport mechanism identified the network latency to be the first limiting factor - with a 6ms RTT we achieved corresponding 155 messages/s in a single stream between 2.5 GHz machines. With 4 streams (still single harvester thread) 595 messages/s were delivered, demonstrating almost linear scalability, while CPU load was still bellow 10%. Therefore LB 7

9 Figure 7. The current (left) and proposed (right) RTM architecture. notifications and the harvester component are unlikely to become a bottleneck of the whole system. In stability tests the harvester also sustained moderate load (600 jobs/h) for several weeks. Recently we ran successful prototype tests of the whole chain, submitting jobs to glite WMS, tracking them by experimental LB, sending notifications to the harvester, and seeing jobs being displayed at RTM maps. Further tests are planned with the aim to release a stable harvester version and to integrate it fully with RTM before the LB 2.0 server is deployed in the EGEE production infrastructure. In this way a smooth migration to LB 2.0 will be achieved. Finally, in LB 2.1 (to be released later in 2009), support for native CREAM jobs is foreseen. Due to clean integration with LB adaptation of RTM to support also CREAM jobs is expected to be easier. 6. Conclusions We have described our development works and operation details of the Real Time Monitor system installed at Imperial College London. We have focused on the RTM architecture but also described the client programs available for general use by Grid users all over the world, enabling them to visualise both the Grid and their job activities within the Grid. The user can easily limit the amount of information available related to their work, i.e. by selecting specific RB or WMS nodes and/or select their only their own VOs. The interest in the RTM is high, we often receive requests from site administrators asking to have their resources included in the RTM monitoring and suggesting changes and improvements. This has proved to be a very successful dissemination tool which has been presented to many visitors worldwide, including heads of states. Reference [1] The EGEE Project. Enabling Grids for E-sciencE. May [2] Andreeva J et..al 2007 Proceedings of Conference on Computing in High Energy and Nuclear Physics (CHEP07), Victoria, Canada. [3] Berkeley Database Information Index: 8

10 May [4] Grid Operation Centre Database, May [5] Lingrand D, Montagnat J, Colling D J and Martyniak J 2009 Analyzing the EGEE production grid workload: application to jobs submission optimization. 14th Workshop on Job Scheduling Strategies for Parallel Processing (JSSPP'09), in conjunction with IPDPS'09 (Springer) [6] RTM webpage, [7] EGEE sites location, as seen by the RTM, ws01.hep.ph.ic.ac.uk/dynamic_information/egee-locations.xml [8] Grid Guide project: [9] ROOT an object-oriented data analysis framework. May [10] Matyska J et al. Job Tracking on a Grid -- the Logging and Bookkeeping and Job Provenance Services. CESNET technical report 9/2007. [11] Andreeva J et al., New strategy for job monitoring on the WLCG scope, J. Physics Conf. Series, this issue 9