Federated data services with abundant compute resources. Thomas C. Schulthess

Transcription

1 Federated data services with abundant compute resources Thomas C. Schulthess 1

2 Sadaf Alam, Lucas Benedicic, Miguel Gila, Cristian Mezzanotte, Colin McMurtrie, Marcel Schöngens, Maxime Montinasso, + Dirk Pleiter, Thomas Lippert, Anna Lührs, Boris Orth + Giovanni Erbacci, Roberto Mucci + Jacob Balma, Jef Dawson, Mark Stavely, Rakhi Chris UC Berkeley: Shane Canon, Doug Jacobsen, Fernando Nicola Marzari, Giovanni Pizz + team; Jeff Muller + Jülich: Katrin Amunz, Timo Dickscheid, Sonja Grün, Alex Peyser + team 2

3 Deep Learning toolkits on Cray XC CSCS DL Toolkit C++ & GPU backend MPI Working on Cray XC Fully CSCS TensorFlow yes no yes yes TensorFlow+MPI yes yes yes in progress MXNet yes no, ext. to use MPI yes in progress Caffe-MPI yes yes yes in progress CNTK yes yes yes in progress Spark no (Java + ext. to use GPUs) no yes in progress Installing a DL toolkit on Cray XC is similar to installing any HPC application few extra libraries are needed to satisfy dependencies Staging a toolkit can be done with SLURM (our resource manager at CSCS) some toolkits (like Spark) require SSH to be available on compute nodes 3

4 DL toolkits on CSCS s Cray XC systems Many toolkits use familiar HPC technology developed in C++ CUDA/GPU-aware libraries HPC filesystems like Luster provide high throughput and data resilience MPI enables access to high performance network such as Cray Aries low latency, high bandwidth Running a toolkit at scale requires a large amount of GPU accelerated nodes HPC platforms are providing this capacity At scale the toolkits face common issues of HPC applications: network and synchronisation boundedness I/O and data access issues HPC systems are very well suited platforms for running Deep Learning workloads at scale 4

5 XC50 supercomputer plus Microsoft s Cognitive Toolkit was used to scale up training 5

6 Scaling CNTK with MPI rank i-1 rank i rank i+1 rank i+2 rank i+3 Samples i-1 Samples i Samples i+1 Samples i+2 Samples i+3 Update Gradient Update Gradient Update Gradient Update Gradient Update Gradient Gradient i-1 Gradient i Gradient i+1 Gradient i+2 Gradient +3 Sum gradients using MPI_Iallreduce Gradient Gradient Gradient Gradient Gradient Update Weights Update Weights Update Weights Update Weights Update Weights 6

7 7

8 8

9 Scaling CNTK with MPI rank i-1 rank i rank i+1 rank i+2 rank i+3 Samples i-1 Samples i Samples i+1 Samples i+2 Samples i+3 Update Gradient Update Gradient Update Gradient Update Gradient Update Gradient Gradient i-1 Gradient i Gradient i+1 Gradient i+2 Gradient +3 Sum gradients using MPI_Iallreduce Gradient Gradient Gradient Gradient Gradient Update Weights Update Weights Update Weights Update Weights Update Weights 9

10 We develop algorithms, we don t have time to deal with C/C++ or MPI a famous computer science colleague working in machine learning 10

11 echoed by many scientists working with data Nishant Shukla (2017) 11

12 Using Jupyter to get our heads around interactive supercomputing Jupyter allow users to integrate development, execution of computation, pre- and post processing with visualisation into one workflow ; share these workflows in a team; and document their work Need to provide an environment where web-based front-end of the notebook is separated from the computation backend run some of the computation on a supercomputer Interactive simply means sub-second response time this requires properly organising data in memory/storage sub-system 12

13 and the service should be up most of the time (like 99+ %) 13

14 Federated Data Pilot Project 14

15 Federated data infrastructure proof of concept Local storage / upload cache Upload service (UFTP) Archive storage Transfer service (UFTP) Download service (UFTP) Active storage Active storage Transfer service (UFTP) Processing 15

16 A few more practical things Scientists, not system administrators will want to manage teams and access to data this poses a fundamental challenge to POSIX style file systems so we have to map between object-store approaches that give users control and POSIX file systems with appropriate performance and in use on supercomputers Software deployment needs to be easy virtual machines; or light-weight VMs: Docker / Shifter as a service 16

17 Architectural Developments Traditional Architecture Research Community CSCS User Data Flow CSCS External Login Access (ELA) Piz Daint Login & Mgmt /store Piz Daint Compute 17

18 Architectural Developments Improved Architecture Based Research Community on External Portal CSCS User Data Flow CSCS Domain Specific Portal Repository access Workflow Manager Does Not Scale External Login Access (ELA) Piz Daint Login & Mgmt /store Piz Daint 18

19 Data Flow Architectural Developments Service Oriented Architecture (SOA) Research Community Domain Specific Portal CSCS User Repository access Workflow Manager Web Services Layer CSCS Security Authentication Authorization User Management Users & Quotas Delegation of PI Data Management Upload/Download Share, Manage Job Automation Submit / Run Jobs Query Queue Capacity Management Shop Reporting Keberos User DB /store Piz Daint DWH 19

20 Supporting Federation using SOA Research Community Domain Specific Portal CSCS User Repository access Workflow Manager Research Community Domain Specific Portal Software services Repository access Workflow Manager Platform services Infrastructure services Infrastructure services 20

21 The data center s concerns of SOA Security & identity management Domain web hosting Data location & transport services Virtual machines Platform Services Data access & management services Compute& location access services Infrastructure Services Container as a service Compute, Storage, Network, Security Domain platform developer services Operations & policy management FENIX compliant Site specific 21

22 22