Cloud Computing The best of both worlds: Conducting compute-intensive HPC applications in a highly flexible cloud environment with low access barriers.
HLRS High Performance Computing Center Stuttgart The High Performance Computing Center of Stuttgart (HLRS) of the University of Stuttgart is the first National Supercomputing Center in Germany and is offering services to both academic users and industry. Apart from the operation of supercomputers HLRS activities include teaching and training in distributed systems, software engineering and programming models, as well as development of new technologies. HLRS is an active player in the European research arena with special focus on Scientific Excellence and Industrial Leadership initiatives. Our Network: HLRS is tightly connected to academia and industry through long term partnerships with global market players such as Porsche and T-Systems, as well as worldwide companies, HPC centres and Universities. Particular attention is given to collaboration with Small and Medium Enterprises (SMEs). Our Infrastructure: HLRS operates a CRAY XC40 supercomputer (peak performance > 7 PetaFlops), as well as a variety of smaller systems, reaching from clusters to cloud resources. Programming Models & Tools Big Data, Analytics & Management Cloud Computing Optimization & Scalability Featured Topics Visualization Exascale Computing Energy Efficiency Services HLRS 4
Our Experience: HLRS has been at the forefront of regional, national and European research and innovation over the last 20 years. During this time, HLRS has participated successfully in more than 90 European research and innovation projects. Our Expertise: HLRS is a leading innovation center, applying software engineering methods to HPC and Cloud for the benefit of multiple application domains such as automotive, engineering, health, mobility, security, and energy. Thanks to the close interaction with industry, the center s capabilities and expertise support the whole lifecycle of simulation covering research aspects, pre-competitive development and preparation for production. The HLRS innovation group, which actively examines and tests new technologies, can bring into projects expertise on leading edge technologies hardware and scale up data analysis techniques. Director HLRS Prof. Dr. Michael Resch HLRS 5
Cloud Computing High-performance computing (HPC) and cloud computing (CC) are both distributed computing infrastructures. While HPC is a specialized method optimized for compute-intensive tasks such as simulation and visualization, clouds are composed of massive amounts of commodity hardware, and thus provide a very large, scalable infrastructure. Typical HPC applications such as large-scale simulations require a specific system architecture that oftentimes cannot be provided by clouds. As a result, applications are not easily portable from HPC to clouds and vice versa and often suffer from poor performance if ported. However, for some users, CC comes with significant advantages over HPC, including the avoidance of a vendor lockin, on-demand provisioning of resources, and capabilities such as cloud bursting and automatic scaling. At HLRS, we try to combine and integrate CC and HPC in order to make HPC more flexible and reduce the barriers for non-experts effectively using HPC infrastructure. In the past, HLRS demonstrated with the European project PaaSage that compute-intensive applications, such as molecular dynamics simulations, can be easily modelled to run in a hybrid HPC/cloud architecture, delegating pre- and post-processing tasks to the cloud. Since virtualization and containerization would increase the flexibility of HPC and at the same time lower the barrier for non-experts, HLRS advances the incorporation of virtualization techniques into HPC. The European project MIKELANGELO demonstrated that virtual machines (VMs) can achieve nearly the same performance as their metal-and-wire counterparts. As a result, we believe that future HPC centers need to allow for virtualization and containerization in HPC; such as by using tools such as vtorque to deploy VMs on HPC infrastructure in a seamless way. Cloud Computing 6
Project Overview FORTISSIMO 2 - Factories of the Future Resources, Technology, Infrastructure and Services for Simulation and Modelling 2 Page 8 MIKELANGELO - Micro Kernel Virtualization for High Performance Cloud and HPC Systems Page 10 EUXDAT - European e-infrastructure for Extreme Data Analytics in Sustainable Development Page 12 Cloud Computing 7
FORTISSIMO 2 Factories of the Future Resources, Technology, Infrastructure and Services for Simulation and Modelling 2 With the advent of computer-aided engineering, small or medium sized enterprises (SMEs) have started to use simulations in order to help their engineers develop innovations as well as create products more cost efficiently. Nevertheless, these complex simulations need High-Performance Computing (HPC) resources to obtain results in a reasonable time, which have not been easily affordable for SMEs so far. Within its logical predecessor the FORTISSIMO project the objective was not only to enable European SMEs to be more competitive by using simulation services running on an HPC cloud infrastructure. The project also offers its stakeholders a one stop shop the FORTISSIMO Marketplace that allows convenient access to such HPC cloud infrastructure and to additional services that are one click away, such as expert support, third-party applications, tools, and helpdesk. In the FORTISSIMO 2 project, the FORTISSIMO Marketplace is en- hanced with advanced HPC cloud services based on High-Performance Data Analytics (HPDA) and coupled HPC simulations. Sensor networks, like weather or traffic flow control, continuously produce high amounts of data. Somewhere in this data lies valuable information, but in order to manage and analyze them in a reasonable time, HPDA becomes indispensable. Coupled HPC simulations are used to study complex models in science, for example, a heart and a blood flow simulation running together for a better understanding of the human circulatory system. But those simulations generate data, which need to be analyzed to obtain value and thus knowledge and expertise. The High-Performance Computing Center Stuttgart (HLRS) is in charge of the FORTISSIMO Marketplace development and operations. This includes extending the Marketplace based on upcoming requirements of the new experiments in FORTISSIMO 2. The project is coordinated by the Uni- FORTISSIMO 2 8
versity of Edinburgh and involves 13 core partners: University of Edinburgh, Scapos, University of Stuttgart, Sicos BW, Intel, Actur, XLAB, CESGA, Gompute, Bull, Atos, SURFsara and CINECA. Project Information Funding Organisation: EU H2020 EU-Funding: 10 Million Runtime: 11.2015 10.2018 Contact Michael Gienger Dr. Bastian Koller Phone: +49 (0) 711 / 685-63824 +49 (0) 711/ 685-65891 E-Mail: gienger@hlrs.de koller@hlrs.de Further Information FORTISSIMO Marketplace: www.fortissimo-project.eu i4ms projects webpage: www.i4ms.eu/projects FORTISSIMO 2 9
MIKELANGELO Micro Kernel Virtualization for High Performance Cloud and HPC Systems During its project runtime, MIKE- LANGELO developed a software stack that enables Big Data and HPC applications to be carried out efficiently in the cloud. In general, the project followed an ambitious approach that improved responsiveness, agility, and security of virtual machines. However, in contrast to traditional research-and-development-oriented projects, MIKELANGELO also demonstrated its achievements in real-world environments. As key partner of the MIKELANGE- LO team, the High-Performance Computing Center Stuttgart (HLRS) contributed to the project s success by creating a job management system that enables virtualised job management for HPC infrastructures. When users run computing jobs in virtual HPC environments, virtual resource administrators need to define the memory, CPUs, software, operating system, and other resources available to the customers, based on the users particular needs. This enables users to utilize a virtual HPC environment including all necessary application components or even operating systems just as they might a traditional HPC environment. As part of MIKELANGELO, HLRS developed a software tool for flexible resource allocation in the cloud that was essential to the project s success. The software, virtualised Torque (vtorque), has been designed not only to schedule traditional batch jobs, but also to manage entire virtual infrastructures. With vtorque users can control an entire application remotely from their own computers, without being restricted to one specific operating system or software. Another challenge that virtualization faces results from the process of data input and output (I/O). This challenge is even more problematic when using HPC applications because of the vast re- MIKELANGELO 10
sources for data processing, management, and storage needed. HLRS s MIKELANGELO partners developed software components to address such I/O challenges for example, by accelerating virtual I/O and establishing high-performance interconnect networks. HLRS staff successfully integrated these software components into vtorque, demonstrating its functionality on both cloud-based and HPC-based system environments. Although cloud environments are known for lower performance compared to HPC infrastructures, the real-world application benchmarks in the cloud resulted in a performance drop of 3% only. Consequently, those numbers show impressively that computing in the cloud becomes interesting also for long-established HPC stakeholders. Project Information Funding Organisation: EU H2020 Runtime: 01.2015-12.2017 Contact Michael Gienger Phone: +49 (0) 711 / 685-63824 E-Mail: gienger@hlrs.de Further Information www.mikelangelo-project.eu MIKELANGELO 11
EUXDAT European e-infrastructure for Extreme Data Analytics in Sustainable Development EUXDAT is an H2020 project with the aim to build an e-infrastructure addressing agriculture, land monitoring and energy efficiency for sustainable development. Agriculture is a key determinant of health, economic and political stability, employment, business and biological ecosystems, and society. Most attention is put on productivity, but in order to address environmental sustainability problems, a global view is needed. Both the challenge and the solution towards solving these sustainibility problems in agriculture relate to the current and future huge amount of heterogeneous data that need to be managed and processed. EUXDAT aims to tackle this challenge. EUXDAT combines existing mature components in order to build up an e-infrastructure addressing agriculture, land monitoring and energy efficiency. A large set of data connectors (UAVs, Copernicus, field sensors, etc.) for scalable analytics will be integrated. An advanced frontend will enable users to develop applications via an infrastructure based on HPC and Cloud, which provides monitoring information, visualization, different parallelized data analytic tools and enhanced data and processes catalogues. HLRS specific role is to provide an infrastructure platform, optimizing resource allocation between the cloud and high performance computing (HPC) infrastructures, EUXDAT 12
their deployment mechanisms, monitoring and management in order to eventually enable Large Data Analytics-as-a-Service. Project Information Funding Organisation: EU H2020 Runtime: 11.2017-10.2020 Furthermore, EUXDAT aims at optimizing the utilization of data and resources, ans evaluating the quality of data in the data management process. With the help of task orchestration the best target for task execution - either an HPC center or a Cloud provider - will be selected. In order to optimize the selection process, monitoring and profiling information will provide the base for decisions about trade-offs related to cost, data constraints, efficiency and resources availability. Contact Michael Gienger Phone: +49 (0) 711 / 685-63824 E-Mail: gienger@hlrs.de Further Information www.euxdat.eu EUXDAT 13
High Performance Computing Center Stuttgart (HLRS) Editor: Lena Bühler, Eric Gedenk, Dr. Bastian Koller University of Stuttgart Nobelstrasse 19 70569 Stuttgart Germany Phone: +49 (0)711 / 685 87 269 Fax: +49 (0)711 / 685 87 209 Design: Janine Jentsch, Ellen Ramminger Picture Credits: Cover and Interior shot: Bohris Lehner for HLRS Back cover shot: Simon Sommer for HLRS Mail: pr@hlrs.de www.hlrs.de HLRS 2018