Agent based parallel processing tool for Big Scientific Data. A. Tsaregorodtsev, CPPM-IN2P3-CNRS, Marseille

Transcription

1 Agent based parallel processing tool for Big Scientific Data A. Tsaregorodtsev, CPPM-IN2P3-CNRS, Marseille IHEP, Beijing, 5 May 2015

2 Plan The problem of the HEP data processing DIRAC Project Agent based Workload Management System Accessible computing resources Data Management Interfaces DIRAC as a service Conclusions 2

3 Big Data Data that exceeds the boundaries and sizes of normal processing capabilities, forcing you to take a non-traditional approach for the treatment Google trends: Clouds Grids Big Data 3

4 Big Data LHC RAW data per year Already in 2010 the LHC experiments produced 13 PB of data That rate outstripped any other scientific effort going on 4

5 Big Data LHC RAW data per year LHC RAW data volumes are inflated by storing derived data products, replication for safety and efficient access, and by the need for storing even more simulated data than the RAW data: ~130 PB overall WLCG data on the Grid 5

6 Data flow to permanent storage: 4-6 GB/sec MB/sec 1-2 GB/sec ~ 4 GB/sec 1-2 GB/sec 6

7 Massive computations: HEP LHC experiments pioneered the massive use of computational grids as a solution to the High Energy Physics Big Data problem 10s of PBytes of data per year 100s of thousands CPUs in 100s of centers 10s of Gbyte/s data transfers 100s of users from 100s of institutions CERN Director General Rolf Heuer about the Higgs discovery: "It was a global effort and it is a global success. The results today are only possible because of the extraordinary performance of the accelerators, including the infrastructure, the experiments, and the Grid computing. Other domains are catching up quickly with the HEP experiments Life sciences, earth sciences, astrophysics, social sciences, etc 7

8 Worldwide LHC Computing Grid Collaboration (WLCG) Distributed infrastructure of 150 computing centers in 40 countries 300+ k CPU cores (~ 2M HEP-SPEC-06) The biggest site with ~50k CPU cores, 12 T2 with 2-30k CPU cores Distributed data, services and operation infrastructure 8

9 Computing grids Grids are providing common infrastructure and rules to integrate multiple computing centers Common computing element access protocols Common user payload scheduling system Common policies and security rules Users are signing up once and have access to multiple computing centers Common monitoring of the user jobs Common accounting of computing resources 9

10 Standard Grid Architecture Centralized Workload Management System Computing sites publish information on their availability Users submit jobs to the central repository The central Resource Broker (RB) pushes user jobs to sites after matching the job/resource compatibility 10

11 Standard grid problems Heavy reliance on the site status information flow Any delays result in suboptimal scheduling decisions Insufficient RB power for large infrastructures Necessity to deploy multiple central (!) RB s Heavy infrastructures required on the sites resources providers Providing timely status, accounting and monitoring information Ensuring per user authentication/authorization Managing community policies via local batch queues parameterization 11

12 Virtualized Computing Resources Since recently computing centers started to provide their resources using cloud technologies Virtualized computing infrastructures Very flexible provides resources adapted to the user needs Operating systems, memory, CPU power, etc. However, large scientific collaborations can have access to multiple computational clouds Dealing with independent clouds separately requires a huge management effort Federating multiple clouds into a single coherent system is necessary to provide a transparent access for users. 12

13 DIRAC Grid Solution DIRAC is developed originally for the LHCb experiment at with the goals: Integrate all the heterogeneous computing resources available to the community Provide solution for both WMS and DMS tasks Minimize human intervention at sites providers of resources Make the grid convenient for the users: Simpler intuitive interfaces Fault tolerance, quicker turnaround of user jobs Enabling Community policies DIRAC has introduced a notion of Pilot Jobs A concept of distributed agents bringing together various disjoint computing resources 13

14 14 DIRAC Workload Management

15 Physicist User Production Manager Matcher Service EGI/WLCG Grid NDG Grid Amazon EC2 Cloud CREAM CE 15 EGI Pilot Director NDG Pilot Director Amazon Pilot Director CREAM Pilot Director

16 WMS: using heterogeneous resources Including resources in different grids and standalone clusters is simple with Pilot Jobs Needs a specialized Pilot Director per resource type Users just see new sites appearing in the job monitoring 16

17 u In DIRAC both User and Production jobs are treated by the same WMS Same Task Queue WMS: applying VO policies u This allows to apply efficiently policies for the whole VO ª Assigning Job Priorities for different groups and activities ª Static group priorities are used currently ª More powerful scheduler can be plugged in demonstrated with MAUI scheduler Users perceive the DIRAC WMS as a single large batch system 17

18 18 DIRAC as a resource manager

19 19 Computing Grids DIRAC was initially developed with the focus on accessing conventional Grid computing resources WLCG grid resources for the LHCb Collaboration It fully supports glite middleware based grids European Grid Infrastructure (EGI), Latin America GISELA, etc Using glite/emi middleware Northern American Open Science Grid (OSG) Using VDT middleware Northern European Grid (NDGF) Using ARC middleware Other types of grids can be supported As long we have customers needing that

20 Clouds VM scheduler developed for Belle MC production system Dynamic VM spawning taking Amazon EC2 spot prices and Task Queue state into account Discarding VMs automatically when no more needed The DIRAC VM scheduler by means of dedicated VM Directors is interfaced to OCCI compliant clouds: OpenStack, OpenNebula CloudStack Amazon EC2 20

21 Standalone computing clusters Off-site Pilot Director Site delegates control to the central service Site must only define a dedicated local user account The payload submission through the SSH tunnel The site can be a single computer or a cluster with a batch system LSF, BQS, SGE, PBS/Torque, Condor, OAR More to come: SLURM, LoadLeveler. etc The user payload is executed with the owner credentials No security compromises with respect to external services 21

22 Examples: DIRAC.Yandex.ru >2000 cores Torque batch system, no grid middleware, access by SSH Second largest LHCb MC production site Standalone computing clusters LRZ Computing Center, Munich SLURM batch system, GRAM5 CE service Gateway access by GSISSH Considerable resources for biomed community (work in progress) Mesocentre Aix-Marseille University OAR batch system, no grid middleware, access by SSH Open to multiple communities (work in progress) 22

23 BOINC Desktop Grids On the client PC the third party components are installed: VirtualBox hypervisor Standard BOINC client A special BOINC application Starts a requested VM within the VirtualBox Passes the Pilot Job to the VM and starts it Once the Pilot Job starts in the VM, the user PC becomes a normal DIRAC Worker Node 23

24 Data Storage and Catalogs Managing data is more complicated than workload management Already existing data in various storage resources must be connected Contents of already existing File Catalogs must be either connected or copied to the DIRAC File Catalog Need for common interfaces to various types of storage and catalog services 24

25 Data Management components For DIRAC users the use of any Storage Element or File Catalog is transparent Up to a user community to choose components to use Different SE types can be mixed together Several File Catalogs can be used in parallel Complementary functionality Redundancy Users see depending on the DIRAC Configuration Logical Storage Elements e.g. DIRAC-USER, M3PEC-disk Logical File Catalog 25

26 Interware: DMS Data Management Multiple types of Storage Elements ( SRM, XRootD, irods, DAV, DIP, ) seen as logical entities for the users SE Proxy service for protocol independence Data replication: FTS3 File Transfer Service Third party Local pipe 26

27 Interware: DMS DIRAC File Catalog Replica Catalog User Metadata Catalog Support for data provenance information Efficient storage usage reports Allow for user quota policies Support for user defined datasets LCG File Catalog Specialized File Catalogs 27

28 Resource provisioning service DIRAC attempts to provide access to all kinds of existing resources and services useful for its users At the same time it provides its own solutions, e.g. catalogs, storage and computing element, transfer service, etc. By design services from different providers can be used together in parallel not necessarily replacing one another Example: use of DFC and LFC together, see below The final choice of the services to use is left to the user 28

29 Interware } DIRAC has all the necessary components to build ad-hoc grid infrastructures interconnecting computing resources of different types. This allows to speak about the DIRAC interware. 29

30 30 Interfaces

31 DIRAC: Secure Web Portal Focus on the Web Portal as the main user tool for interactions with the grid Intuitive desktop application like interface Ajax, Tornado, ExtJS Javascript library Monitoring and control of all activities User registration, proxy upload User job monitoring and manipulation, downloading results Data manipulation and downloads DIRAC Systems configuration and management Secure access Standard grid certificates Fine grained authorization rules 31

32 32 Web Portal examples

33 Application Web Portals Specific application portals can be built in the DIRAC Web Portal framework Community Application Servers DIRAC RESTful interface Language neutral Suitable to use with portals written in Java, PHP, etc Other interfaces include Extensive Python API A rich set of command line tools ( >200 commands ) Available on UNIX-like systems 33

34 34 DIRAC Users: large communities

35 LHCb Collaboration 35 Up to 100K concurrent jobs in ~120 distinct sites Equivalent to running a virtual computing center with a power of 100K CPU cores Further optimizations to increase the capacity are possible Hardware, database optimizations, service load balancing, etc

36 Experiments: Belle II Combination of the non-grid, grid sites and (commercial) clouds is a requirement 2 GB/s, 40 PB of data in 2019 Belle II grid resources WLCG, OSG grids KEK Computing Center Amazon EC2 cloud First production run is done Thomas Kuhr, Belle II 36

37 Belle II DIRAC Scalability tests Hideki Miyake, KEK 37

38 Community installations ILC/CLIC detector Collaboration, Calice VO Dedicated installation at CERN, 10 servers, DB-OD MySQL server MC simulations DIRAC File Catalog was developed to meet the ILC/CLIC requirements BES III, IHEP, China Using DIRAC DMS: File Replica and Metadata Catalog, Transfer services Dataset management developed for the needs of BES III CTA CTA started as France-Grilles DIRAC service customer Now is using a dedicated installation at PIC, Barcelona Using complex workflows Geant4 Dedicated installation at CERN Validation of MC simulation software releases Alpes Laser Company using dedicated DIRAC setup for simulation of the laser hardware Mostly using commercial Amazon resources 38 DIRAC evaluations by other experiments LSST, Auger, TREND, Daya Bay, Juno Evaluations can be done with general purpose DIRAC services

39 39 DIRAC as a Service

40 DIRAC as a service DIRAC client is easy to install Part of a usual tutorial DIRAC services are easy to install but Needs dedicated hardware for hosting Configuration, maintenance needs expert manpower Monitoring computing resources is a tedious every-day task Small user communities can not afford maintaining dedicated DIRAC services Still need easy access to computing resources Large grid infrastructures can provide DIRAC services for their users. 40

41 Several regional and university campus installations in France Complex maintenance Joint effort to provide France-Grid DIRAC service Hosted by the CC/IN2P3, Lyon, T1 center Distributed team of service administrators 15 VOs, ~100 registered users Mostly biomed applictions 41 France-Grid DIRAC service

42 Multi-VO DIRAC service in IHEP IHEP Computing Center is the main resource provider for the BES experiment The BES Collaboration decided to use DIRAC to build its distributed computing project Developed a good expertize level in developing and managing DIRAC services The BES dedicated DIRAC service is being extended to support other user communities CEPC, Juno, Daya Bay, Non-HEP communities? Reuse of acquired experience Single Administration team Minimization of the user support and maintenance effort 42

43 43 Conclusions

44 Conclusions The computational grids are no more something exotic, they are used in a daily work for various applications Agent based workload management architecture allows to seamlessly integrate different kinds of grids, clouds and other computing resources DIRAC is providing a framework for building distributed computing systems and a rich set of ready to use services. This is used now in a number of DIRAC service projects on a regional and national levels Services based on DIRAC technologies can help users to get started in the world of distributed computations and reveal its full potential 44