Efficient Access to a Cloud-based HPC Visualization Cluster

Size: px

Start display at page:

Download "Efficient Access to a Cloud-based HPC Visualization Cluster"

Christine Hudson
6 years ago
Views:

2 Agenda 1. Data Science and HPC challenges 2. Technologies that improve remote visualization performance 3. A solution approach 2

4 Data Science Challenges Work from a remote office Lots of data gathered Difficult to move data Manage applications on HPC cluster Where to run visualization Process is complicated and slow 4

5 Challenge - Moving the Data Too much data! Copying TBs over the Internet Variable speeds Sometimes lacks reliability Wait hours to find out results 5

6 Technologies - Moving the Data RoCE and Dedicated Circuit RDMA over Converged Ethernet Low latency Dedicated Circuit Reliable QoS High-performance throughput Specialized routers 6

7 Challenge - Managing the Application Cluster may not have application Difficult to build HPC applications correctly, that is HPC cluster complexity Distribute workloads GPUs for visualization Job scheduling 7

8 Technology - Managing the Application Containers Data scientist can deploy to cluster Crafted for specific HPC cluster Enterprise: containers tend to be generic HPC: Custom containers fit HPC needs 8

9 Technology - Managing the Application Containers - Docker Large base of existing images Easy to create custom images May need a developer Focus is on DevOps Manage friction between development and cloud operations 9

limits features HPC file systems and network Older

10 Technology - Managing the Application Containers - Docker issues for HPC Root escalation Securing root limits features HPC file systems and network Older versions of Linux Job scheduling HPC distributed file systems 10

11 Technology - Managing the Application Containers - Singularity Focus on portability Security: same user inside and out Designed for HPC HPC file systems and networks Built-in GPU support singularity.lbl.gov Integrates MPI and job scheduling Import Docker images 11

create wrf-workflow.img % singularity import wrf-workflow.

12 Technology - Managing the Application Containers - Singularity Building container from Docker % singularity create wrf-workflow.img % singularity import wrf-workflow.img docker://wx/wrf Deploying the container URI or Singularity Hub % scp wrf-workflow.img cluster://apps/wrf Running the container % singularity run /apps/wrf/wrf-workflow.img 12

14 Challenge - Visualizing the Results Visualization on workstation (GPU) Reach into HPC cluster for results Package and download results Client software management Error prone, slow, and complex 14

15 Technology - Visualizing the Results Real-time, interactive GUI Desktop workflows 3D visualization Remote visualization from browser No plugin or application needed Designed for HPC desktop apps High-end workstation performance on-demand GPU cloud environment 15

16 Technology - Visualizing the Results NVIDIA GRID technology H.264 quality with low latency GPU Accelerated Applications Quadro 5000 graphics, K10 GPU 4K resolution Multi-display Secure access through HTTPS 16

18 Challenge - Building the Solution Scientists need to focus on science Complex, changing technologies Hard to manage HPC environment Custom solution means support 18

19 Technology - Building the Solution Technologies available today On-premise or HPC cloud POD - On-Demand HPC cloud Scyld Cloud Manager Cloud Workstation penguincomputing.com/solutions/pod-hpc-cloud/ penguincomputing.com/products/software/ 19

DIGITAL TRANSFORMATION IN HEALTHCARE

DIGITAL TRANSFORMATION IN HEALTHCARE Delivering Cost-Effective, High-Value Healthcare with NVIDIA Virtual GPU Solutions Year after year, the healthcare industry is faced with challenges revolving around