EXDCI Overview Transversal vision and strategic prospective

Transcription

1 Overview Transversal vision and strategic prospective François Bodin WP4 28 May 2018 The project has received funding from the European Unions Horizon 2020 research and innovation programmed under grant agreement No

2 Introduction, a CSA in a rapidly evolving context Since beginning a lot has happened As planned, new FETHPC projects and CoEs IPCEI European Open Science Cloud EuroHPC Joint Undertaking Exascale still a moving target Now systems 50x to 100x faster than 2017 on real apps in 2022/2023 Pathways to convergence data and compute Extension of the use of supercomputers (e.g. urgent computing) Increased international competition China first machine in

3 Starting Point (EESI2) Scientist teams Pre-processing Data assimilation Compute intensive post -processing Simulation results Data from sensors Data mining Long-term storage In-situ analysis loop 3 Simulation data storage From «Les big data à découvert», CNRS Éditions, 2017 Data extraction Data refinement

4 Where we are now Borrowed to Dan Reed (BDEC 2018 presentation) 4

5 Today: GAFAM Market Dominated Borrowed to Dan Reed (BDEC 2018 presentation) 5

6 Outline of the Talk Aspects of the paradigm shift Petascale to Exascale Supporting Complex Scientific Workflows Addressing the paradigm shift in Disruptive technology SME analysis Analyzing our ecosystem Conclusion & lessons learnt 6

7 Aspects of the Paradigm Shift Petascale to Exascale 7

8 Petascale to Exascale Petascale to Exascale transition is raising many issues Not only related to technology Not happening in isolation In a context of scientific (observational) data deluge Well summarized in the USA National Strategic Computing Initiative (NSCI) NSCI seeks to drive the convergence of compute-intensive and data-intensive systems We are potentially on a paradigm change denoted Exascale but meaning computing generation transition 8

9 Petascale to Exascale cont. Peta-Exa transition is not similar to Tera-Peta transition 9 This is a disruption mainly due to parallel model issues (compute and IO) And the need to deal with large amounts of data from multiple sources (scientific instruments, simulations) Some questions are Can one platform fit all? Is the Cloud a relevant solution? How to move data around (or not)? How to integrate Big Data technologies? How to manage resources?

10 Why Peta-Exa is Not Similar to Tera-Peta Transition? The main Tera-Peta transition was performed before during the Giga-Tera transition Adaption of codes to distributed memory machines Tera to Peta was smooth and with minimum (side-) effects for most HPC users Data issue is changing the game for Peta-Exa New software stack and algorithms Questions the discovery process (e.g The Fourth Paradigm) Data analytics and machine learning Data localization 10

11 What Exascale is Not Exascale == flops of interest for a small community Such as LQCD and field based on embarrassingly parallel methods (e.g. Monte-Carlo) Exascale transition for most people is not about the next increment in machine features The next generation of machines is likely to create a practice and organization disruption It is easy to compute anywhere (c.f. PRACE) but moving data around is (very) slow, if feasible Adherence to a system (including storage and networking) is likely to increase 11

12 Exascale-Wise Applications Characterization* 1. Workload 2. Workflow 3. Code 4. Scalability 5. Operating System 6. I/O 7. HPC Community 8. Hardware 9. Visualization 10.Interactivity 11.Data management and analysis 12.Impact on Science/Society *Computer science point of view 12

13 Exascale Transition Impact on Codes Tera-to-Peta Peta-to-Exa Off-the-Shelf ISV in charge Not addressed (Market?) In-House Languages Update of the codes, but no significant effort compare to Giga-to-Tera Fortran, C/C++, OpenMP, Cuda, OpenACC, OpenCL Too many technologies for an in-house team. Will need to add software engineers, etc. Fortran, C/C++, OpenMP, OpenACC, OpenCL, DSL, interpreted languages, task support, Runtime Accelerator support Runtime must handle more resources Sustainability No significant changes More specialized codes (e.g. DSL), code architecture rendered obsolete Complexity Portability Code refactoring to make it accelerator friendly Not simple, but achievable with careful design Heterogeneous software stack to address the data and task management Very dependent on workflow management and application architectures Performance Retuning (expensive in some cases) possible Tuning is going to be Hell (too many dimensions), energy tuning?* 13 *1w/year 1, 10% of 20MW/Y 2MW/Y 2M /Y ~20 persons/y

14 Exascale Transition Impact Example with Workflows Tera-to-Peta Peta-to-Exa Complexity Code coupling More multiphysics, multiphase models, data assimilations, data analytics, edge computing, Heterogeneity Mostly homogeneous Mix of data analytics and simulation, heterogeneous bricks Localization All in one system Data may come from large scientific instruments, or a large number of small instruments I/O constrained Solvable issue Cannot move the data around, not sure it can be solved Allocation Batch mostly Batch, interactive (guided simulation and analysis), (soft) real-time* (visualization, ) *Big data assimilation for Extreme-scale NWP, Takemasa Miyoshi 14

15 Aspects of the Paradigm Shift Supporting Complex Scientific Workflows 15

16 Combining Edges, Data Centers, Supercomputers-1 Complex workflow and data logistic to map onto the set of systems Bottleneck frontiers The edge Intermediate nodes (CDN, data centers, etc.) Strongly coupled node (Supercomputers, data centers, etc.) How to manage/program applications uniformly across systems? 16

17 Combining Edges, Data Centers, Supercomputers-2 Function of a system node is by destination not by nature Energy / throughput tradeoff, compute vs. communicate Bandwidth and latency between nodes can vary Interoperability is the goal A supercomputer node is not different from an edge node (capabilities only differ) A supercomputer is defined by a set of nodes that are strongly connected by high bandwidth, low latency links Interoperability between nodes helps to create complex workflows that includes all sorts of tasks (i.e. sensing, simulation, analysis, visualization) There are no frontiers between the core supercomputers and their edges or intermediate nodes or the storage nodes Complex workflow can be distributed in a uniform way Storage is just high capacity (e.g. NVM) reliable nodes 17

18 Workflow Model as a Spanning Layer Allows for optimizing cost, latencies, energy and QoS for complex workflows Need a user model What is a user? Can a user have a global credential? How do you link users and resources? Require an extended resource management model Compute, storage, network Composition rules (e.g. synchronizations) Security and privacy Should have a common meta-data referential Data localization, origin, replication, Should apply to all (sets of) nodes 18

19 An Example of Workflow: Air Quality Measurement and Analysis (Including Urgent Computing / closed-loops) New users / applications Storage heterogeneity Communication heterogeneity Computation heterogeneity CPU/hours per simulated day (not an exascale issue) 19 Current AQMO project in Rennes, FR

20 Going Beyond: Incremental Progress Very difficult to extend current infrastructure concepts to fit the future needs Sources of data are growing very fast (in and outside the HPC community) Many future applications rely on complex distributed workflows Conservatism / conflicts of interests of stakeholders is an issue Global end-to-end performance should be the focus With the assumption that data cannot be moved around but need to be accessed from anywhere 20

21 Addressing the Paradigm Shift in 21

22 Addressing the Paradigm Shift in Disruptive technology Understanding which technologies may disrupt practices SME analysis in HPC, how/can SMEs create more economical value? Analyzing our ecosystem What should be changed at ecosystem level? 22

23 Disruptive Technologies Phase 1: analysis of three cross-cutting issues in conjunction with nine disruptive innovation strategies and for each of them proposes recommendations (e.g. NVM, Byte addressable I/O) Phase 2: Focus on Quantum Computing - Promote the use of high-end HPC systems of the PRACE infrastructure to emulate QC and provide tool and environment to allow developer to run and develop applications on the QC emulator - Hire expert people on QC from top institutions and train HPC people by visiting exchanges and other instruments A 2014 prototype of a Google qubit (0.6 cm by 0.6 cm) known as a transmon 1, based on superconducting circuits. From aps.org. Google's quantum computing test will use 49 updated versions of these qubits. - Propose coordination actions at the European level 23

24 Addressing the Paradigm Shift in Disruptive technology SME analysis Understanding our ecosystem 24

25 EU projects as cradle for new technologies and start-ups Supportive environment Partnership already in place (project consortium) Shared risk (via EU financing) Mont-Blanc 1-3 Deep/Deeper/ Deepest Kaleao and Zeropoint Technologies (emerged from Euroserver project) but nevertheless * Projects reported only on very few pieces of technology * Projects seem not aware of existing help for start-ups, nor much sensibility for helping SMEs of their project Only few submissions to FET Launchpad calls * Survey amonst the CoEs and FETHPC projects in March

26 Input from SMEs and start-ups Goal Get the perception of SMEs developing technologies and start-ups on the market, hurdles, specific problems -> via interviews (15) Outcome 1. Reluctance to innovation on the market We don t say its novel HPC centres need above all to comply with their clients needs 2. Difficult to get into the market No one ever got fired for buying intel Pre-condition: have been already part of a top-500 machine Circumvent HPC: penetrate market via applications or via similar markets 26

27 Input from SMEs and start-ups (cont.) 3. Public procurement Financial requirements for bidding more difficult to meet for an SME than for a large company Need for pre-financing can hinder SMEs form bidding Procurement schemes such as PPI not much used 4. Connection to the ecosystem (e.g. via EU projects): important but financially difficult for SME ( we cannot spare 0.5 ETP for a project ) 27

28 SME Recommendation Example R6: Unlocking innovations in EU-projects by stimulating projects spin-off in order to benefit of the existing collaboration (within a running R&I project) and by redirecting funds of the R&I project towards a spin off for exploiting Cupboard IP 28

29 Addressing the Paradigm Shift in Disruptive technology SME analysis Analyzing our ecosystem 29

30 An Ecosystem Cartography (-wise) Showing impact on the Ecosystem How do we connect stakeholders Graph representation Two types of nodes Events that generate a common production SME workshop SRA, PSC, EsD BDEC 2017 Technical Workshops Stakeholders, CoE, FET HPC Edges represent the participate to relationships PSC Cerfacs Eocoe 30

31 Entities Active in Related Activities About 80 entities 31

32 A Map of the Ecosystem as Connected by Includes about 260 entities 32

33 Main Conclusion on the Ecosystem Focus and organize the current efforts in a way that is closer to an integrated industrial project in order to ensure a successful delivery of European Exascale level sustainable systems capable of serving a convergence based scientific discovery process 33

34 Conclusion and Lessons Learnt 34

35 Conclusion and Lessons Learnt 1. HPC is experiencing a paradigm shift HPC vision has changed dramatically since the beginning of 2. The ecosystem is getting wider and more heterogeneous 1. BDEC, Big Data, Cloud, IoT, 2. HPC a strategic tiny part of it 3. And will have to evolve at high speed 3. Difficulties for the emergence of EU HPC SMEs and startups still need a proper response 4. EU R&D organization can still be improved 35

36 Conclusion and Lessons Learnt Exascale transition is a holistic issue that Cannot ignore the data issues Nor the market structure / GAFAM domination EuroHPC the main EU vehicle toward Exascale More EU technologies in HPC systems? Pre-Exascale systems first EU Exascale systems 2019 China, 2021/2022 Japan and US -2 will continue -1 s work 36

37 FEAT Future Emerging Art and Technology Initiative supported by the commission, the artists engaging with the HPC projects participated to one workshop The semiotic of supercomputers (2017), Špela Petrič and Miha Turšič

38 Backup slides 38

39 Understanding our Ecosystem (-wise) How does the Ecosystem contributes to the recommendations? What is the coverage of actions? What type of stakeholders do we have? 39

40 Innovation, SMEs, start-ups and tech transfer Glimpse at the ecosystem (european level) In : A. Get input form the community 1. From the projects (FETHPC and CoEs) 2. From SMEs and start-ups B. What can we do on ecosystem level C. Outlook 40

41 SME Recommendations R1: SME should team up with main bidder for big procurement is encouraged as a way to go especially for large procurement R2: HPC centres should have innovation oriented projects and HPC systems R3: The SME instrument should be better promoted amongst the HPC community and if necessary better adapted to the HPC sector R4: Mechanism to build trust and personal relationship amongst SMEs and Start ups should be encouraged R5: Partnership of SMEs/Start ups with HPC centres and large vendors shall be encouraged 41