Data Management for the RedisDG Scientific Workflow Engine

Transcription

1 Data Management for the RedisDG Scientific Workflow Engine Per3S (Performance and Scalability of Storage Systems) DDN Leila Abidi, Souha Bejaoui, Christophe Cérin, Jonathan Lejeune, Yanik Ngoko, Walid Saad Laboratoire d Informatique de Paris Nord Jan 30, Leila Abidi, Souha Bejaoui, Christophe Cérin, Jonathan Data Management Lejeune, Yanik for the Ngoko, RedisDG Walid Scientific Saad Workflow Engine

2 2 / 30 Table of Contents 1 Elements of (local) context 2 Problem definition 3 Contributions 4 Conclusion - Future work 2 Leila Abidi, Souha Bejaoui, Christophe Cérin, Jonathan Data Management Lejeune, Yanik for the Ngoko, RedisDG Walid Scientific Saad Workflow Engine

3 Digital platform: 3 / 30 1) 2) 3) 4) Request Software install Reserve nodes Send job Get results SYS ADMIN USER MAGI S-CAPAD CUMULUS 1) VM install 2) Softfare install inside VM 3) Use Software 4) Network of VMs (soon) CIRRUS TODAY (SaaS) 3 Leila Abidi, Souha Bejaoui, Christophe Ce rin, Jonathan Data Management Lejeune, Yanik for the Ngoko, RedisDG Walid Scientific Saad Workflow Engine

4 4 Leila Abidi, Souha Bejaoui, Christophe Cérin, Figure: Jonathan Data Management SlapOS Lejeune, Yanik architecture for the Ngoko, RedisDG Walid Scientific Saad Workflow Engine 4 / 30 Disruptive industrial projects (french technology) SlapOS cloud ( open source Started in VM was an option (not mandatory); Based on Linux LXC ; A conceptual view based on 3 ingredients: One ERP (to manage the catalog of applications and the customer relations); A model for the deployment; Nodes;

5 5 / 30 Disruptive industrial projects (french technology) The Q.rad is a smart and connected digital heater, fusion of an electrical heater and a high-performance computing server; Q.rad are located at home... and produce heat by computation; Figure: Qarnot Computing platform 5 Leila Abidi, Souha Bejaoui, Christophe Cérin, Jonathan Data Management Lejeune, Yanik for the Ngoko, RedisDG Walid Scientific Saad Workflow Engine

6 6 / 30 Innovative industrial projects (french technology) Fully automated, the Q.ware platform is in charge of : Optimal computing node selection (Q.rads, data center, IaaS, cloudlets) Client payload/container boot sequence Input data distribution and output results collection Processors frequencies adjustment Job management (progress control, logging, automatic recovery) Industrial talk for EuroPar 2016 conference: Heating as a cloud-service, a position paper, Yanik Ngoko (Qarnot Computing) scheduling scientific issues. 6 Leila Abidi, Souha Bejaoui, Christophe Cérin, Jonathan Data Management Lejeune, Yanik for the Ngoko, RedisDG Walid Scientific Saad Workflow Engine

7 7 / 30 1 Elements of (local) context 2 Problem definition 3 Contributions 4 Conclusion - Future work 7 Leila Abidi, Souha Bejaoui, Christophe Cérin, Jonathan Data Management Lejeune, Yanik for the Ngoko, RedisDG Walid Scientific Saad Workflow Engine

8 8 / 30 Scientific computing before/after the cloud 8 Leila Abidi, Souha Bejaoui, Christophe Cérin, Jonathan Data Management Lejeune, Yanik for the Ngoko, RedisDG Walid Scientific Saad Workflow Engine

9 9 / 30 Scientific computing before/after the cloud 9 Leila Abidi, Souha Bejaoui, Christophe Cérin, Jonathan Data Management Lejeune, Yanik for the Ngoko, RedisDG Walid Scientific Saad Workflow Engine

10 10 / 30 Scheduling DAGs MONTAGE: experiments on a large instance 9423 input files (including the intermediary files) and the workflow generates 2889 files (including the intermediary files). 10 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

11 11 / 30 State of the art Scheduling: Theoretical foundation: to think about what we have to do. Sometimes too abstract. Often related to the performance metric only in a context of everything is known in advance; Heuristics: to solve concrete problems... those with influencing parameters identified by some experiments; Our heuristics are natural. Goes to simplicity. The novelty: ways we manage them and interaction between components; 11 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

12 12 / 30 State of the art Scheduling: Theoretical foundation: to think about what we have to do. Sometimes too abstract. Often related to the performance metric only in a context of everything is known in advance; Heuristics: to solve concrete problems... those with influencing parameters identified by some experiments; Our heuristics are natural. Goes to simplicity. The novelty: ways we manage them and interaction between components; Tools/Middleware: BOINC, Condor, XtremWeb, BonjourGrid; OpenAlea for plant phenomics ( pradal.pdf): DIET ( id=551) SWIFT ( D. Talia survey ( 12 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

13 13 / 30 Our general thesis Building systems for heterogeneous and highly dynamic environments we need to be compliant with: 1. a publish-subscribe layer for the orchestration of the components of the system; 2. a set of opportunistic strategies for allocating work/tasks that are also based on the publish-subscribe layer; 3. a small number of software dependencies for the system and the ability to deploy the system and its applications on demand. This point is of particular interest in this paper and we promote the easy to use, and systems that can be deployed without a system administrator. 13 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

14 14 / 30 1 Elements of (local) context 2 Problem definition 3 Contributions 4 Conclusion - Future work 14 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

15 15 / 30 Contributions (a step forward) WaaS (Workflow as a Service): Computing infrastructure easy to deploy (on demand) in a cloud; Easy to use for everyone (expert level not required); Based on the RedisDG framework: The initial framework has been formally proved; To play with opportunistic scheduling; To play with the Publication/Subscription paradigm for the interactions between the component of the RedisDG system; Specifying a distributed system according to the Publication/Subscription paradigm is challenging (and not frequent... even with Google Pub-Sub :-) 15 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

16 16 / 30 Opportunistic scheduling Many modern computing platforms (clouds, desktop grids, and volunteer-computing projects) exhibit extreme levels of dynamic heterogeneity (availability and relative efficiencies); When an event occurs decide on the best think to do based on the knowledge available at this time; 16 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

17 17 / 30 Opportunistic scheduling Many modern computing platforms (clouds, desktop grids, and volunteer-computing projects) exhibit extreme levels of dynamic heterogeneity (availability and relative efficiencies); When an event occurs decide on the best think to do based on the knowledge available at this time; Event Service Publisher Publisher Publisher Publisher Publish Publish Storage and management of subscriptions Subscribe() Notify() Unsubscribe() Subscribe Unsubscribe Notify Subscriber Notify() Subscriber Notify() Subscriber Notify() Subscriber Notify() 17 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

18 18 / 30 Platform-oblivious vs Platform aware Opportunistic Scheduling Theoretical point of view (Arnold L. Rosenberg, EuroPar 2016) Opportunistic dag-execution via Platform-Oblivious Scheduling: One always benefits computationally with dag-structured workflows by enhancing the likelihood of having as many eligible chores as possible. Such scheduling enhances the likelihood of having work available as (advantageous) resources become available, hence being able to exploit resources opportunistically. 18 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

19 19 / 30 Platform-oblivious vs Platform aware Opportunistic Scheduling Theoretical point of view (Arnold L. Rosenberg, EuroPar 2016) Opportunistic dag-execution via Platform-Oblivious Scheduling: One always benefits computationally with dag-structured workflows by enhancing the likelihood of having as many eligible chores as possible. Such scheduling enhances the likelihood of having work available as (advantageous) resources become available, hence being able to exploit resources opportunistically. In the concrete life: network bandwidth, hierarchical architecture or heterogeneous computing nodes (CPU, GPU, FPGA) Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

20 20 / 30 Interest and implications of our proposal From the interest point of view: volunteering architectures are very interesting for HTC problems: However, HTC often generate a lot of data whereas voluntering architectures are based on low bandwidth networks minimizing transfer costs is very relevant. 20 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

21 21 / 30 Interest and implications of our proposal From the interest point of view: volunteering architectures are very interesting for HTC problems: However, HTC often generate a lot of data whereas voluntering architectures are based on low bandwidth networks minimizing transfer costs is very relevant. From the implications point of view: Do we need to always wait for the response of all workers? Do we need to transfer data to a central node at the end of the calculation? What about replication of jobs? Time to deploy the infrastructure? As a service? 21 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

22 22 / 30 The problem we solve Problem definition: given a DAG and a of queue for requests, find an allocation such as a performance criteria (time, energy, load... ) is minimized/maximized; In this paper: data aware approaches for opportunistic scheduling in order to minimize the execution time; 22 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

23 23 / 30 The problem we solve Problem definition: given a DAG and a of queue for requests, find an allocation such as a performance criteria (time, energy, load... ) is minimized/maximized; In this paper: data aware approaches for opportunistic scheduling in order to minimize the execution time; Implementation: RedisDG framework 23 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

24 24 / 30 Centralized:Decentralized management Centralization versus Decentralization 24 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

25 25 / 30 Heuristics for minimizing the transfer of data Input Number (IN) = score(w j, T i ) = p Pred(i,j) card(i T i O p) Input Size (IS) = score(w j, T i ) = p Pred(i,j) f (I Ti O size(f ) p) 25 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

26 26 / 30 Heuristics for controlling a fairness principle Based on observations and to reduce the effect of monopolizing the tasks by fewer workers, accentuated by both previous heuristics: Fair Root Input Number (FRIN): for tasks on the first level; based on IN; Fair Root Input Size (FRIS): for tasks on the first level; based on IS; Fair: generalization of the fair distribution to all the levels of the tasks graph: Balance the number of tasks performed by each worker independently of the data emphasize that the reduction of the execution time in the FRIN and FRIS heuristics is not only due to equity but to the Combination of equity and data 26 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

27 27 / 30 Experimental results (sample) Homogeneous physical machines; (1+4/1 1+4/3 for the Docker case) MONTAGE workflow (NASA); Testbed: Large-scale and versatile testbed for experiment-driven research in all areas of computer science, with a focus on parallel and distributed computing including Cloud, HPC and Big Data. C-FIFO D-FIFO IN IS FRIN FRIS FD Data (s) Exe (min) 8:39 7:40 6:21 6:44 6:01 6:28 7:30 27 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

28 28 / 30 1 Elements of (local) context 2 Problem definition 3 Contributions 4 Conclusion - Future work 28 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

29 29 / 30 Conclusion - Future work Data aware scheduling and opportunistic scheduling schema; Implementation in the RedisDG framework; Experiments; Work done (since the submission): Introduction of Docker containers + Introduction of heterogeneity of hardware; Introduction of decisions policies based on the load or the energy consumption or the fairness ; Current work: Multicritera approaches (including the availability of nodes) HPC in the cloud; Predictor(s): load, network bandwidth... according to ML techniques; Modelization of the coupling of different execution models (one node of the DAG may be unfold dynamically) - Provenance/Data life cycle (ActiveData). 29 Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine

30 30 / 30 Data Management for the RedisDG Scientific Workflow Engine Per3S (Performance and Scalability of Storage Systems) DDN Leila Abidi, Souha Bejaoui, Christophe Cérin, Jonathan Lejeune, Yanik Ngoko, Walid Saad Laboratoire d Informatique de Paris Nord Jan 30, Leila Abidi, Souha Bejaoui, Christophe Cérin, Data Jonathan Management Lejeune, for Yanik thengoko, RedisDGWalid Scientific Saad Workflow Engine