Cloud-Based HPC and HPDA. Dr. Alfred Geiger, Dr. Christian Simmendinger

Transcription

1 Cloud-Based HPC and HPDA Dr. Alfred Geiger, Dr. Christian Simmendinger

2 AGENDA T-Systems SfR and HWW An integrated Cloud-Concept for the Challenges of the Future Dynamic HPC-Capacity and Capability HPC Cloudifier and Performance Engineering High Performance Data Analytics, Data Discovery and Machine Learning Quantum Annealing

3 T-SYSTEMS SOLUTIONS FOR RESEARCH GmbH Company Facts Scientific Workplace Services Founded: Campus Services Employees: ca.160 Locations all over Germany Europe-wide Delivery Bremen Braunschweig Berlin Communication Services Security Services User & Identity Management 100% Subsidiary of T-Systems International History Founded in 1999 as a joint venture of Deutsches Zentrum für Luft- und Raumfahrt e.v. and debis Systemhaus Köln Bonn Göttingen Lampoldshausen Scientific Desktop Services Scientific Computing Design and Services HPC and HPDA Platforms Customers Public Research Organizations Industrial R&D Departments LegendE Hauptstandorte Weitere Standorte Stuttgart Garching Ulm Weßling / Oberpfaffenhofen Software Development Support Scientific Data Management Joint Research 3

4 Höchstleistungsrechner für Wissenschaft und Wirtschaft GmbH Company Facts Founded in 1995 as a Joint-Venture for the dynamic provisioning of HPC-Resources for science and industry. Resources provided by HLRS, KIT/SCC and T-Systems SfR HPC-Cloud Shareholders Karlsruher Institut für Technologie KIT (18.75%) Universität Stuttgart (18.75%) Land Baden-Württemberg (12.5%) T-Systems International GmbH (20%) T-Systems SfR GmbH (20%) Porsche AG (10%) Dynamic HPC Provisioning (HPC Cloud) 4

5 AGENDA T-Systems SfR and HWW Integrated Cloud-Concept for the Challenges of the Future Dynamic HPC-Capacity and Capability HPC Cloudifier and Performance Engineering High Performance Data Analytics, Data Discovery and Machine Learning Quantum Annealing

6 OTC-Centric Provisioning-Concept for Scientific- and Technical Computing Access to any kind of Scientific- and Technical Computing via the Open Telekom Cloud (OTC) Even external systems look for the user like specialized nodes. Advantage: Infrastructure for Accounting, Network, IdM, Security, Data-Management) can be used from OTC. Europe-wide Availability New Computing paradigms can easily be integrated either as part of the OTC (GPU, FPGA) or remote (High-End HPC, High Performance Data Analytics, Quantum-Computing, Neuro-Computing...) Any kind of engineering and scientific workflow can be mapped Certified End-to-End Security-Concept. 6

7 Open Telekom Cloud as Backbone for a Science- and Engineering Cloud User GPU Computing High Energy Physics Media Streaming FPGA Computing Genome Assembly (Health) Seismic (Oil and Gas) Machine Learning 10 GE Open Telekom Cloud Data-Management and Analysis Network-Access Security Accounting IdM Throughput Computing Pre- & Postprocessing Virtual Front-Ends Géant Quantum Computing Optimization Telecommunications 3D printing Portfolio (Finance) Radiation-Therapie (Health) Cryptoanalytics Machine Learning HPC/HPDA Numerical Simulation Engineering Weather/Climate Data Discovery High-Speed Analytics 7

8 AGENDA T-Systems SfR and HWW Integrated Cloud-Concept for the Challenges of the Future Dynamic HPC-Capacity and Capability HPC Cloudifier and Performance Engineering High Performance Data Analytics, Data Discovery and Machine Learning Quantum Annealing

9 Dynamic Provisioning in Scientific- and Technical Computing Capacity 0-64 Cores per Job Distributed I/O Accounting based on Reservation or Usage GPUs, Remote Graphics, VDI Virtualised Provenance and Monitoring Different Flavors Capability Cores per Job Scalable I/O Bare Metal Pay per use or per reservation Different Provisioning Models Open Telekom Cloud (OTC) HPC-Cloud 9

10 HPC Platforms of HWW Very large scalable Systems Variety of Systems The optimal solution for each problem Economic Advantages operational synergies 10

11 T-Systems HPC ON-DEMAND RESERVED, DEDICATED SECURITY Strong 2-factor authentication, VPN encryption End-to-end customer specific hardware Dedicated file systems Dedicated Pre-Post processing infrastructure Dedicated Remote visualization nodes. Immutable customer specific OS images. Reservations in partitioned network segments. Customer specific user adminstration. SUPPORT FOR COMPLEX WORKFLOWS Seamless integration of complex workflows. Meta scheduler integration License server integration 11

12 AGENDA T-Systems SfR and HWW Integrated Cloud-Concept for the Challenges of the Future Dynamic HPC-Capacity and Capability HPC Cloudifier and Performance Engineering High Performance Data Analytics, Data Discovery and Machine Learning Quantum Annealing

13 HPC-Cloudifier and Performance Engineering Objectives Transfer of Workflows from local HPC-Environments to distributed environments based on cloudconcepts Improvement of Scalability to achieve best in class performance on highly scalable systems Dynamic, application specific, runtime optimization Definition of application and customer specific templates for cloud-environments Improvement on overall costs per simulation in cloud-environments 13

14 Showcase PowerFLOW Objectives Project-Objectives The goal of the project is the improvement of scalar and parallel efficiency of PowerFLOW (EXA) in HPC-Cloud Environments based on the optimisation of the runtimeenvironment. Reduction of turnaround-times, TCO 14

15 Showcase Powerflow Summary and Outlook Summary In a dynamic and adaptive runtime-environment it is possible to achieve a performance-gain of 15-20% for the PowerFlow solver. Outlook A source-code optimisation of PowerFlow w/r to MPI and scalability seems to be useful and doable. An adaptive runtime-environment opens new perspectives for the optimisation of other solvers in cloud-environments. 15

16 AGENDA T-Systems SfR and HWW Integrated Cloud-Concept for the Challenges of the Future Dynamic HPC-Capacity and Capability HPC Cloudifier and Performance Engineering High Performance Data Analytics, Data Discovery and Machine Learning Quantum Annealing

17 Data Analytics Data Analytics We know, what we are looking for Data can be split up (Map) on the base of a distributed Filesystem (e.g. Hadoop Filesystem) Analytics runs independent on the different chunks of data. Results are merged together in a reduction operation (Reduce) Infrastructure: Virtualized Cloud-Environments with platformservices (e.g. Cloudera) Real-Time Analytics and Predictive Analytics Examples: Decision Support, Cyber Physical Systems (IoT) Requirements High-Speed I/O (Fast Networks (Bandwidth and Latency), Scalable and parallel I/O (Lustre (intel), GPFS (IBM), BeeGFS (FhG), Isilon,...) based on Infiniband-coupled I/O-Clusters In-Memory Processing 17

18 Data Discovery We don t know, what we are looking for Examples Medical Pesearch: Relations between disease-syndroms and genetic dispositions. Search for correlations between usage-patterns and failures in consumer-products. Social and Societal analytics. Analysis of criminal or terrorist networks Methods Use of graph-based methods (information is represented as node, correlations as edges of the graph). As the graph is the result of the analysis, data cannot be split up and has to be treated in a global adress space. Analytics performed on closely coupled HPC-Systems Platforms Large Memory nodes in cloud-environments + Spark GraphX HPE/sgi UV + Spark GraphX Cray Urika + CGE or Spark GraphX 18

19 HPDA-Platform: Cray Urika GX Local storage Global Lustre 19

20 HIGH FREQUENCY INSIGHTS SERIOUS AGILITY PERVASIVE SPEED System is tuned for demanding analytics Dynamic workflows for comparison and adaptation Cray Graph Engine for fast, complex, pattern matching In-memory semantic graph database based on W3C standards such as RDF & SPARQL Tuned for serious speed on noisy data with complex relationships Cray Graph Engine Outperforms Average Test Speedup of 280X > 200x Faster LUBM25K with >3 Billion Triples Spark GraphX vs. Cray Graph Engine Results in Seconds on Urika-GX system > 400x Faster 3 X Faster than prior Urika generation on complex queries

21 AGENDA T-Systems SfR and HWW Integrated Cloud-Concept for the Challenges of the Future Dynamic HPC-Capacity and Capability HPC Cloudifier and Performance Engineering High Performance Data Analytics, Data Discovery and Machine Learning Quantum Annealing

22 Complexity Not all Problems can be solved with Binary Computers Compute-Power Non Solvable (e.g. decryption, optimisation) Actual Limit of Compute-Power based on linear scalability Solvable (e.g. Numerical Simulation) Problem-Complexity 22

23 Simulated Annealing on Binary Computer vs. Quantum Annealing Quantum-Tunnelling Due to their complexity, most optimisation problems can not be solved exactly. Approximative Solutions can be found on the base of Heuristics, so called Simulated Annealing Only Local Optima can be found based on Simulated Annealing. Based on the Principles of Superposition and Entanglement, a Quantum-Computer can represent all possible Solutions simultaneously and select the absolute Optimum and thus the Exact Solution via Energy-Considerations. To exclude the Distortion of the Solution based on the Injection or Subtraction of Energy, adiabatic Quantum Computers have to be shielded from any kind of Radiation and they have to be operated in Superconductivity mode. 23

24 HPC and HPDA ON-DEMAND SERVICE CONTACTS DR. ALFRED GEIGER SCIENTIFIC COMPUTING CURIESTRASSE STUTTGART PHONE: MOBILE: MAIL: ALFRED.GEIGER@T-SYSTEMS.COM ROLAND DOHMEN DEAL ENTRY GATE AM GUT WOLF 9a AACHEN PHONE: MOBILE: MAIL: ROLAND.DOHMEN@T-SYSTEMS.COM INTERESTED? ANY QUESTIONS? PLEASE DO NOT HESITATE TO CONTACT US. 24