The EPIKH Project (Exchange Programme to advance e-infrastructure Know-How) Introduction to glite Grid Services

Transcription

1 The EPIKH Project (Exchange Programme to advance e-infrastructure Know-How) Introduction to glite Grid Services Fabrizio Pistagna Beijing, China Asia Joint CHAIN / EPIKH School for System Administrator

2 is the official Middleware for grid computing developed and adopted by EGEE EGEE is a series of projects funded by European Commission it connects more than 70 institutions in 27 European countries What is glite after prototyping phases in 2004 and 2005, convergence with the LCG-2 (LHC Computing Grid) distribution was reached in May 2006 when glite 3.0 was realeased and became the official middleware of EGEE project Lightweight Middleware for Grid Computing 2

3 What does not? Somehow my application just run faster Any users can access to any resource Users can rely of a huge amount of software, libraries What does? What glite does? Applications will run faster only if properly developed and thinked for Grid environment runtime ( trivial parallelization, MPI ) Provides access to computational/storage/other resources, accordingly to defined Policies and access rights Provides sharing of resources ( CPUs, Disks, Sensors... ) Allows the creation of Virtual Organizations ( people, groups ) and let them cooperate 3

4 glite Grid scenario Users of different organizations geographycally distant, are grouped in Virtual Organizations that share computational, storage and other resources among other VOs......resulting in COLLABORATION! Members of a Grid can be part of different VOs at the same time 4

5 glite Grid components services User Interface (UI) is the user access point to the Grid site Workload Management System (WMS) is a set of services which are in charge of finding the suitable resources for the submitted job and scheduled jobs Virtual Organization Membership service (VOMS) is the way glite improves the management of authentication and authorization to the Grid resources. The VOMS allows to their own members to define different access rights to VO s resources Logging and Bookkeeping (LB) keeps track of user job execution in terms of events 5

6 glite Grid components services Information System (IS) maintain data related to available grid resources and their health statuses, in order to optimize resource exploitation Computing Element (CE) is the computational resource, the entry point to a cluster handled by a job queue management system (TORQUE/PBS, LSF, CONDOR, SGE) Worker Nodes (WN) are the machines where jobs are really executed Storage Element (SE) allow to manage Grid files and offer a mechanism to locate them easily for users and jobs 6

7 Very simple Job Life Cycle 7

8 glite Grid services 8

9 glite Grid services Grid services can be grouped into five high level sets of services Grid Access Security Information System & Monitoring Job Workload Management System Data Management 9

10 glite Grid Access There are two ways for access to a Grid site managed by the glite Middleware Native APIs : job management ( Java, Python, C++ ) Data management ( Java, C++ ) Resource discovering Provided Command Lines Utils A set of command lines utilitities that are distribuited together with the glite-ui packages 10

11 glite Grid Security Authentication based on X.509 PKI infrastructure The user receives a certificate from a Certification Authorities (CA) The user has to subscribe to a VO He/She connects to the UI ( typically via SSH ) He/She creates the user proxy, all Grid services will use this proxy to identify the user Authorization The user has to subscribe to a VO The VO establish the user rights In any Grid service it will be verified if the user belongs to the VO and VOMS assigns the proper access rights to the user VOMS ( Virtual Organization Membership Service ) is a service that keeps track of the members belonging to a VO supports MyProxy ( stored proxies ) grants users authorization to access the resources at VO Level 11

12 glite Grid Security service : VOMS It extends the proxy certificates with info on VO membership, groups, roles ( Administrator, normal user, student ), capabilities Each VO has a database containing group membership, roles and capabilities information for each user User contacts VOMS server requesting his authorization informations If request is legitimate, VOMS server sends signed authorization info to the client Client includes them in a proxy certificate 12

13 In order to submit a job to the glite Grid Workload Management contact the Workload Management System infrastructure, users This is the glite service that performs all tasks required to execute user s jobs, without exposing the user to the complexity of the Grid It is composed by different services The WMProxy, is the access point to the WMS itself The Workload Manager is the most important component The Workload Manager accepts and satisfy the requests for job management coming from its clients, delivers them to an appropriate Computing Element (CE) for the execution 13

14 glite Grid WM service : WMS JDL file User Interface W M P R O X Y Workload Manager Task Queue Job Adapter Resource Broker Submission Process 14

15 Computing Element CE is some set of computing resource localized at a site CE is responsible to manage the queue of jobs to execute Is generally composed by Grid Gate: which acts as a generic interface to the cluster LocalResourceManagementSystem or batch-system: OpenPBS, Condor, Maui/Torque, LSF Cluster: a collection of Worker Nodes, the nodes where the jobs are run CE may be used by a generic client: when an end-user interacting directly with it by the Workload Manager: which submits a given job to an appropriate CE found by the matchmaking process 15

16 glite Grid WM service : CE Job management through the WMS provides many benefits compared to direct job submission to the CE WMS can manage multiple CEs is able to forward the job requests to the one which better satisfies a set of requirements WMS provides a Global job tracking facility using the LB service WMS supports complex job types, which can not be handled directly by the CEs Dag, collections, parametric WMS handles job failures if a Job aborts, WMS automatically re-submit it to the matched CE 16

17 glite Grid Information System What is an IS? Is simply a System to collect informations on the state of the Grid Resources Why we need an IS monitoring Grid site resources? To discover resources of the Grid and their nature To check health status of the resources To help the workload more efficiently How? Simply monitoring all the published Grid resources and publishing fresh data on their state Who will use IS? User searching specific resources for their activity Workload Management System Other monitoring systems 17

18 glite Grid Information System: BDII The IS architecture used in glite is based on the Berkeley Database Information Index (BDII) it stores information at VO level Site GIIS ( Grid Index Information Server ) stores information at site level GRIS ( Grid Resource Information Server ) stores information at resource level 18

19 glite Grid Data Management Data management services are at least Storage back-end resources types ( Disk Pool Manager, Mass Storage System, dcache ) Storage Resource Manager interface ( SRM ) It is a Middleware interface application, that makes standard data management operations between SEs of different resource type, transparent to the end user These data management operations include: File transfers Space reservation Renaming of files File directory creation Transfer services to transfer files in and out of a SE Globus GridFTP protocol (gsiftp) Globus-url-copy, https, ftp protocols 19

20 glite Data Management: LFC LCG File catalog simply keeps track of the location of a file in the Grid in a logical way so that a file will be accessible from anywhere The identification of files on the storage element is done through the use of different identifiers LFN Logical File Name: An alias created by the user to refer to some item of data e.g. lfn:/grid/disk-pool/simulations/run01/outputdata.01 GUID Globally Unique Identifier: A non-human readable unique identifier for an item data e.g. guid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6 SURL Storage URL or Physical Name: The location of an actual piece of data on a storage system e.g. srm://grid009.ct.infn.it/dpm/ct.infn.it/gilda/output10_1 TURL Transport URL: Describes access protocol for physical retrieval of files e.g. rfio://lxshare0209.cern.ch//data/alice/ntuples.dat 20

21 glite Data Management: LFC LFC mantains mappings between LFN, GUID, SURL 21

22 glite Grid services 22

23 Any questions? 23