HPC IN THE CLOUD Sean O Brien, Jeffrey Gumpf, Brian Kucic and Michael Senizaiz

Transcription

1 HPC IN THE CLOUD Sean O Brien, Jeffrey Gumpf, Brian Kucic and Michael Senizaiz Internet2, Case Western Reserve University, R Systems NA, Inc 2015 Internet2

2 HPC in the Cloud CONTENTS Introductions and Background Sean O Brien Internet2 Campus Viewpoint Jeffrey Gumpf Case Western Reserve University Industry Viewpoint Brian Kucic & Michael Senizaiz R Systems NA, Inc. [ 2 ] 2015 Internet2

3 Background Key Benefits Democratization Elasticity Flexibility Key Challenges Network Cost and pricing model Data lock-in

4 Acting as a community to overcome challenges Internet2 network Community negotiated contract terms IRU model Data egress waiver Sharing experience and expertise [ 4 ]

5 Analytics Use Cases Jeffrey Gumpf, Case Western Reserve University [ 5 ]

6 There have been tremendous advances in parallelized algorithms and distributed storage in recent years [ 6 ]

7 Driven by the MapReduce paradigm introduced by Google in 2004 and the Google File System introduced in 2003 [ 7 ]

8 Allowing algorithms and data storage to be scaled and parallelized across large clusters of computers [ 8 ]

9 An open source ecosystem called Hadoop was created around those paradigms [ 9 ]

10 A decade later there now is a very rich ecosystem that exists for parallelized computation that includes Hadoop and much, much more [ 10 ]

11 This ecosystem makes highly parallel computation and data manipulation easily accessible to researchers [ 11 ]

12 Much of modern research analytics, capable of processing 100s of petabytes of data across 10,000s of computers, is now built on top of the Hadoop ecosystem and covers almost every research area including Humanities, Engineering, Medicine, and the Sciences [ 12 ]

13 The ecosystem supports a wide range of high performance computation including database queries, machine learning, data mining, search, matching, high volume stream processing, processing of massive graphs, genetics algorithms, among many others [ 13 ]

14 Providers of cloud infrastructure services have embraced the this new ecosystem making it possible to quickly assemble, scale, and then disassemble clusters to support this software ecosystem [ 14 ]

15 This relieves the researchers and IT support staff of many of the tasks surrounding the acquisition, building, operation, and maintenance of the environment [ 15 ]

16 This lets researchers devote more of their resources to research rather than dealing with infrastructure [ 16 ]

17 There is a growing cadre of companies that support and further develop the analytics software ecosystem to help it grow and to better solve problems its users struggle with [ 17 ]

18 This provides researchers with an ever growing portfolio of new and powerful tools to help them solve research problems. Tools which would be difficult and resource consuming to create themselves [ 18 ]

19 Most of the major cloud vendors support the analytical ecosystem with highly scalable ondemand infrastructure architectures [ 19 ]

20 Clusters of computers, from two to tens of thousands, can be brought into full operation within minutes and then quickly shut down when the task ends [ 20 ]

21 Cloud infrastructure vendors often support a wide variety of instance types with varying memory sizes, disk drives, networking speed, and even GPUs in some cases [ 21 ]

22 Users only pay for what they use [ 22 ]

23 Some of the vendors offer ways to reduce costs, such as interruptible instances at much lower cost (sometimes less than 20% of the normal instance cost) [ 23 ]

24 The modern, open source, analytical software environment is designed to be fault tolerant, automatically recovering from failures of components without special effort on the part of the user [ 24 ]

25 Many commercial software vendors have modified their proprietary software to include support for open source analytical ecosystems [ 25 ]

26 Scalability, low cost, and ease of use make a compelling case for high performance analytical computing in the cloud [ 26 ]

27 R Systems Brian Kucic and Michael Senizaiz [ 27 ]

28 R Systems R Systems NA, Inc. provides technical expertise and HPC cluster resources to the academic and commercial research communities through the R Systems brand and the Dell HPC Cloud Services partnership. Our mission is to minimize clients time to solution and maximize effectiveness and efficiency, adding value to every HPC result. [ 28 ]

29 Usage Models No barriers to entry Dedicated or Exclusive Use Models Fast Implementation Custom Service Level Agreements Pay-as-you-go or Utility Use Models Overflow and Peak Shaving Capabilities No Multi-Year Affiliation Fees No Minimum Usage Requirements Privately Owned;; including all infrastructure [ 29 ]

30 Automated Compute/Storage No barriers to entry AUTOMATED Offering in Q4 of 2015 Self Service Compute Partitioner Custom high performance storage backend Infiniband Interconnected HPC Resources Pay only for what you use [ 30 ]

31 HPC IN THE CLOUD Sean O Brien, Jeffrey Gumpf, Brian Kucic and Michael Senizaiz Internet2, Case Western Reserve University, R Systems NA, Inc 2015 Internet2