Agile Computing on Business Grids

Transcription

1 C&C Research Laboratories NEC Europe Ltd Rathausallee 10 D St Augustin Germany Junwei Cao Agile Computing on Business Grids An Introduction to AgileGrid June 2003

2 Agile Computing on Business Grids An Introduction to AgileGrid 1 Introduction Grid computing [Cao2003b] will lead to a global computing infrastructure, which will provide an ideal basis to achieve agile computing [Cao2003c] In this document, detailed technical issues are discussed on how to build agile computing services on business grids (AgileGrid) A java package, JAG (Java for Agile Grids), is designed to provide an all-in-one solution to AgileGrid This document is a work in progress and feedback is encouraged 2 AgileGrid: Agile Computing on Business Grids Agile computing [Cao2003c] is proposed to be a new computing paradigm that allows greater variety and individuality in computing services Agile computing cannot be achieved for mass customization of computing services without a global computing infrastructure that enables mass production of computing power The grid is an ideal candidate for such a global computing infrastructure, especially future grids for commercial usage [Cao2003b] Technical issues related to enable the grid have been addressed in the research of grid community These include: resource management, information services, data management, security, programming models, QoS and adaptation support, performance evaluation, monitoring and measurement, portals and PSEs, and so on [Cao2003b] Some of these are potential to be utilized to enable agile computing and are introduced below Workflow Management Workflow and process management has been a research topic in business management and other areas for many years Since a grid application could also be a collection of loosely coupled activities, workflow is considered as a programming model for grid applications Related research issues include workflow specification language, workflow engine/enactor, workflow simulation, workflow scheduling, workflow monitoring, workflow editor, and so on Our initial experience to implement workflow management in a grid computing environment can be found in [Cao2003a] AgileGrid requires that grid services should be customized according customers QoS requirements If a grid service is implemented as a flow of utilization of other grid services, dynamic service federation will be essential to achieve agile grid computing That means service instances should be created after customers QoS requirements arrive, which provide the maximum flexibility to adapt to various requirement In this way grid services are 2

3 customized and each customer is served with a different service instance that is considered to be the best one to meet his requirement Dynamic service federation is essential to achieve agile grid computing and has to be implemented efficiently to fit into the real time situation Several topics below are related to the implementation of dynamic service federation Performance Prediction Numerous methodologies have been developed for modeling, analysis, evaluation and predication of HPC applications, including benchmarking, analytical modeling, simulation, and monitoring Our previous experiences on performance modeling of HPC applications and its use for grid computing QoS support can be found respectively in [Cao2000] and [Cao2002a] Performance predictive information is also quite useful for decision making processes of dynamic service federation in agile grid computing Example performance metrics include execution time, cost, memory and disk usage Analysis and measurement based performance prediction is more likely to be used in agile grid computing scenarios Service Discovery Current service discovery mechanism and protocols developed in grid computing are centralized Distributed service discovery is more addressed in P2P systems In our previous work, a P2P-like service advertisement and discovery mechanism is designed in an agentbased grid computing system [Cao2001, Cao2001b] AgileGrid requires dynamic service discovery capabilities Service discovery processes should not rely on a single point service registration and discovery service Services should be able to find each other in a self organizing way using some distributed mechanisms and protocols Dynamic service federation requires service discovery to be processed efficiently and rapidly Scheduling Resource scheduling and workload balancing is important to achieve high performance computing on clusters or multiprocessors In a grid computing environment, meta/co/super schedulers are used for higher level control of multiple local grid schedulers Our previous work on cluster scheduling and grid load balancing can be found in [Cao2002] and [Cao2003] AgileGrid brings new challenges on more flexible scheduling processes For example, mapping workflow activities efficiently onto multiple choices of grid services according to multiple performance metrics is a difficult task, because it is not only an issue of advanced scheduling algorithms but also relies on complicate policies In agile grid computing, scheduling becomes interactive processes among the customer, the customized service and other grid services The scheduling should be processed according to customers preferences if required Automatic Negotiation While there is little research on the use of automatic negotiation techniques for grid computing scheduling and QoS support This has been identified as a useful technique to achieve agile grid computing A negotiation can be described using communication protocols or workflow descriptions Related issues need to be refined further 3

4 Customer Integration AgileGrid requires that the customer to be the most important partner of the whole computing processes The customer cannot be simplified as a request sender and result collector The customer may be required to participate in all the AgileGrid processes, eg business management, QoS negotiation, service selection and federation, and so on The customer may have their own preferences on how to use the grid Customer integration needs software support to interface customer to software systems For example, predefined usage patterns can be managed by customer software agents, which are intelligent and autonomous software entities that interface customers to software systems Related issues need to be refined further Security AgileGrid requires more flexible interactions among grid services, which may lead to security problems For example, a customized grid service has to carry out flows of computing tasks by consuming other grid services on behalf of the customer Current grid security infrastructure may be not enough to achieve this It may be necessary to find a good tradeoff between flexibility and security Security issues need to be refined further All of above issues have to be addressed to achieve the AgileGrid and corresponding software systems have to be designed according to some predefined architecture Since grid computing infrastructures are following a service-oriented approach and the emerging Open Grid Service Architecture (OGSA) is becoming the de facto standard architecture for grid computing, AgileGrid will also be implemented within the OGSA framework The implementation of AgileGrid will end up with the development of some high level grid services on top of grid core and base services (Or directly web services based? This is also an issue to be decided in future) 3 JAG: Java for Agile Grids As Globus is an open source implementation of the OGSI, JAG is considered to be an all-inone Java solution to AgileGrid and provides the following services: Grid Workflow Services These services provide interfaces for runtime handling of workflow instances These may include the creation and deletion of a workflow instance, retrieving information about workflows (eg partners and their properties), changing partners of workflow activities, adding and removing activities, and so on These services allow modification to deployed workflows and are essential for the implementation of dynamic service federation for agile grid computing For example, if the QoS negotiation result for a new customer request requires that one of current workflow partners should be replaced by another grid service, the changing partner service can be called to perform this without un-deploying the whole workflow (Should workflow services definitions be related to a specific workflow language, eg BPEL4WS or GSFL?) Grid Performance Services These services provide general support to performance prediction of the corresponding grid service, eg performance data management (Should performance services be related to a specific prediction approach or algorithm? If so, it seems the most suitable one is the statistical analysis of historical information) 4

5 There may be many other services required to support AgileGrid, eg grid negotiation services, grid proxy services (I will add here details in future) While Globus can be used for mandatory implementation of grid services to guarantee the interoperability, at this moment, JAG is designed to be a package that developers can choose to implement their applications with only necessary services Interoperability of agile grid computing applications can only be achieved if the AgileGrid service infrastructure can be standardized in future Software developers can build grid applications directly on top of the OGSI using toolkits like GT3, however, in this case, all high level services we mentioned above aiming at the AgileGrid will has to be implemented as application-specific services JAG is motivated by the generalization of these high level services and aims to provide more services to facilitate grid application development Even if a user does not totally agree with our vision of agile computing, JAG will be still a useful Java package that can provide more grid application enabling support References [Cao2000] [Cao2001] [Cao2001b] [Cao2002] J Cao, D J Kerbyson, E Papaefstathiou, and G R Nudd, Performance Modelling of Parallel and Distributed Computing Using PACE, in Proc of 19 th IEEE Int Performance, Computing and Communication Conf, Phoenix, USA, pp , 2000 J Cao, D J Kerbyson, and G R Nudd, Performance Evaluation of an Agent- Based Resource Management Infrastructure for Grid Computing, in Proc of 1 st IEEE/ACM Int Symp on Cluster Computing and the Grid, Brisbane, Australia, pp , 2001 J Cao, D J Kerbyson, and G R Nudd, High Performance Service Discovery in Large-Scale Multi-Agent and Mobile-Agent Systems, Int J Software Engineering and Knowledge Engineering, Special Issue on Multi-Agent Systems and Mobile Agents, Vol 11, No 5, pp , 2001 J Cao, S A Jarvis, S Saini, D J Kerbyson, and G R Nudd, ARMS: an Agent-based Resource Management System for Grid Computing, Scientific Programming, Special Issue on Grid Computing, Vol 10, No 2, pp , 2002 [Cao2002a] J Cao, S A Jarvis, D P Spooner, J D Turner, D J Kerbyson, and G R Nudd, Performance Prediction Technology for Agent-based Resource Management in Grid Environments, in Proc 11 th IEEE Heterogeneous Computing Workshop, in conj with 16 th IEEE Int Parallel & Distributed Processing Symp, Fort Lauderdale, FL, USA, pp 86-99, 2002 [Cao2003] [Cao2003a] J Cao, D P Spooner, S A Jarvis, S Saini, and G R Nudd, Agent-Based Grid Load Balancing Using Performance-Driven Task Scheduling, in Proc 17 th IEEE Int Parallel & Distributed Processing Symp, Nice, France, pp 49-58, 2003 J Cao, S A Jarvis, S Saini, and G R Nudd, GridFlow: Workflow Management for Grid Computing, in Proc 3 rd IEEE/ACM Int Symp on Cluster Computing and the Grid, Tokyo, Japan, pp ,

6 [Cao2003b] [Cao2003c] J Cao, From Scientific Grids to Business Grids - The State-of-the-Art of Grid Computing, Agile Computing Series White Paper, C&C Research Laboratories, NEC Europe Ltd, 2003 J Cao, Agile Computing Enabling Mass Customization of Computing Services, Agile Computing Series White Paper, C&C Research Laboratories, NEC Europe Ltd,