HPC in Industry: Recent Studies from Intersect360 Research and U.S. Council on Competitiveness. Addison Snell

Transcription

1 HPC in Industry: Recent Studies from Intersect360 Research and U.S. Council on Competitiveness Addison Snell

2 Intersect360 Research in th year of HPC analyst business Covering high performance data center markets, including traditional HPC and supercomputing, highperformance enterprise, cloud, big data, and hyperscale New hyperscale advisory service Weekly podcast, This Week in HPC Contributing editors to The Next Platform, market research partners of TOP500.org, media partnerships with HPCwire, insidehpc, others Sponsors of HPC Advisory Council

3 What We Count As HPC High-Performance Technical Computing Applications in science and engineering Includes verticals such as manufacturing, pharma, oil and gas High Performance Business Computing Non-scientific business applications Includes verticals such as financial services, entertainment, retail, transportation

4 A New Market: Hyperscale Users whose computational needs, application domains, and business model demand a web-scale infrastructure that incorporates scalability and performance. In general, includes organizations that have applications requiring thousands of servers to operate and require certain economies of scale to be viable. Examples include web search, content hosting, cloud hosting, online retail, social media, and others. Intersect360 Research has counted Ultrascale Internet as an HPC segment since 2007 We have taken this segment out of our HPC methodology and are now tracking it in a separate advisory service

5 HPC vs. Hyperscale Similarities Application infrastructure that is distinct from general enterprise Applications are arbitrarily scalable Top tier users push boundaries and influence technology throughout industry HPC Differences Hyperscale Primary focus is performance Large jobs Is still HPC on only a few nodes there is entry-level HPC Largest supercomputers are > $100 million Primary focus is scalability Many jobs At lower levels it is just enterprise computing no entry-level Largest hyperscale installations are >$1 billion

6 About the Solve Report Extreme Computing and U.S. Competitiveness Copyright 2014

7 Solve Report: Key Findings 1. Computing Capacity 2. Software Scalability 3. Public Partnership Copyright 2014

8 Importance of HPC Research Results & Analysis HPC helps us improve our products and come up with new products: protect the environment, make sure our products are safer, that they are more efficient, more durable, and that they have unique functionality...

9 Research Results & Analysis How challenging to justify HPC Investment?

10 Research Results & Analysis Projected Need for Most Challenging Applications 10x 68% 100x 57% 1,000x 37%

11 Research Results & Analysis Barriers to Achieving 10x Greater Scalability

12 Research Results & Analysis Mix of Software Usage

13 Changes in the CFD Landscape CFD Software by Source by Survey Year

14 Download & Share Report Available at: Follow the Council on Twitter

15 HPCAC Working Group Surveys Surveys from three Council working groups: HPC Access HPC Skills HPC Software Targeting U.S.-based commercial HPC-using organizations: 64 qualified respondents Range of vertical markets, budget sizes, experience, etc.

16 Access Survey: Key Findings There is a need for HPC resources Industry is mostly unaware of federal HPC access programs Where awareness exists, it comes most commonly from conferences and events. One user specifically suggests better outreach, like showing up to industry events. Importance of capacity, capability, technical resources, software all about equal Intellectual property concerns are the biggest potential blockers in evaluation

17 Need for More HPC Resources 15% said they have access to adequate HPC resources already 40% said they could consume over 100% more than they have currently

18 Awareness of Federal HPC Programs Best awareness: LLNL Innovation Center LANL Inst. Computing NSF XSEDE NSF Blue Waters

19 Why Not?: Intellectual Property Among HPC users who chose not to utilize a federal HPC program, 56% said that concerns about intellectual property and proprietary information control were an extremely significant factor in their decision.

20 Notable Suggestions (Direct Quotes) In our case the software is starting to be the limiting factor. the cost per CPU has come down, but the per seat cost of the software has not. Being able to leverage that would be huge. Easier access to desired software licensing I don't want to have to search [for], get, [and] sign NDAs to get software set up. Improved access to commercial licenses. Less formality/bureaucracy once a relationship is established, instead of applying for every project. Issues around export control and IT security are paramount. The amount of internal effort required to get approval to access and use outside computing resource can be substantial. Contracting/legal aspects are still case-by-case, and there is no well-known pattern. This piece takes too long. It should be a much lighter initiative, like buying a cloud service. Some of our models contain our customers information, and they will not let that reside on hardware outside our immediate control. All parties in our industry [are] overworked and focused on the geophysical research they are working on. It seems the scientists/engineers are not encouraged to look outside the box, and so they don't look for other resources to help with their problems. This may be due to concerns about intellectual property. Something to consider is not just the IP of the product, but the method of analysis. We would not want such things shared with our competitors if we went to an outside resource for support. There are trade secrets that would need to be protected also.

21 Skills Survey: Key Findings The most training is required not in analyzing / validating, but in finding mistakes, improving, rolling out new methods Retraining experienced employees takes less effort; similar pattern to what s hardest There is a recognized need for significant education or cultural change, especially for executives

22 How Difficult to Close Skill Gaps?

23 Skill Gaps The biggest gap I see with new graduates is that they generally understand what these tools are capable of but have no idea how to use them or even which tool is appropriate for a certain type of job. In a world where multicore and heterogeneous computing is default (considering mobile and generalpurpose to be ubiquitous), then parallel computing should be the default behavior, but it's still taught as a special skill. This has to change.

24 Comp. Literacy of Exec. Decision-Makers ~40% in each category need training or significant culture change

25 Software Survey: Key Findings Current/projected barriers to deploying software: Cost of scaling software licenses 59% rated 4/5 for today 47% rated 4/5 over next five years Cost/difficulty of developing applications in-house 47% rated 4/5 for today 47% rated 4/5 over next five years Demonstrating ROI 44% rated 4/5 for today 32% rated 4/5 over next five years Lack of optimism that things will improve dramatically Best collaboration partner may be ISVs Collaboration is valued, but challenging to implement

26 Software Issues Given the challenges faced in scaling software, it is relevant to examine the prospects for collaboration and partnership to ameliorate the difficulties, but here there are challenges as well. Among the possible challenges to collaboration, the biggest fear is the loss of control of intellectual property. The vast majority of respondents rated intellectual property agreements to be important or extremely important in pursuing collaborations. But behind this perhaps-obvious assertion lie other potential pitfalls, as other touchpoints, including cost (e.g. of proprietary software partnerships), access to knowledge and expertise, and the time to begin the project, were all also rated as important by at least half the respondents. This presents a dilemma: that collaborations must be undertaken quickly, at low cost, with access to knowledge and expertise, while simultaneously protecting intellectual property as the most important criterion. Not surprisingly, although collaborations together with other industry participants would be potentially valuable, this type of partnership is viewed as the most challenging to be undertaken. Government agencies, too, are not viewed as promising, rated as the least valuable type of partnership, and the second-most challenging to conduct. If partnerships are to succeed, the best payoff may come from dealing directly with software partners themselves. Software providers were rated as the most valuable or effective potential collaboration partners, and simultaneous as the least challenging to work with.

27 HPC Software Agree / Disagree Increased collaboration between the U.S. Government and software companies, academia, and industrial users would facilitate HPC software development. 88% agree There need to be more/better venues for working collaboratively on underlying software tasks, such as mesh generation or model validation, to benefit multiple applications and industries. 82% agree My competitors in other countries (non-u.s.) are at an advantage because of government programs in those countries that support better software development. 50% agree The United States is on par with or ahead of other countries in developing effective, scalable software applications for future supercomputing architectures. Only 9% disagree Even if the U.S. is not behind in the race, it is clear that more can still be done in order to drive leadership.

28 HPC in Industry: Recent Studies from Intersect360 Research and U.S. Council on Competitiveness Addison Snell