HPC Analytics in the Era of Big Data J. Robert Michael, PhD Sr. Software Engineer St. Jude Children s Research Hospital

Transcription

1 HPC Analytics in the Era of Big Data J. Robert Michael, PhD Sr. Software Engineer St. Jude Children s Research Hospital

2 Outline HPC at St Jude what are we doing? Automated image analysis what are the issues? Data mining GPFS what is your hardware doing? Data mining your apps what is your software doing?

3 Our Mission Statement The mission of St. Jude Children s Research Hospital is to advance cures, and means of prevention, for pediatric catastrophic diseases through research and treatment. Consistent with the vision of our founder Danny Thomas, no child is denied treatment based on race, religion or a family's ability to pay.

4 Cost to family: $0 Families never receive a bill from St. Jude for treatment, travel, housing or food because all a family should worry about is helping their child live.

5 Cryo EM Genomics Microscopy Biostatistics HPC at St Jude cores ~53 TB RAM ~8.8 PB GPFS 16x Nvidia K80 s Structural Biology Radiation Oncology Developmental Neurobiology

6 Automated Image Analysis: Issues The issues: Image analysis is often done on (Windows based) workstations Data is large (and growing!) and workstations are feeling the crunch

7 Automated Image Analysis: Solution? The issues: Image analysis is often done on (Windows based) workstations Data is large (and growing!) and workstations are feeling the crunch The solution: Scale image analysis on HPC

8 Automated Image Analysis: More issues The issues: Image analysis is often done on (Windows based) workstations Data is large (and growing!) and workstations are feeling the crunch The solution: Scale image analysis on HPC The (other) issue: Imaging software is not well suited for Linux How can you annotate multi TB image sets for validation on a desktop? File conversion is a pain!

9 Automated Image Analysis: Idea The solution: Scale image analysis on HPC

10 Automated Image Analysis: Idea File conversion The solution: Scale image analysis on HPC

11 Automated Image Analysis: Idea File conversion The solution: Scale image analysis on HPC Image analysis (at scale)

12 Automated Image Analysis: Idea File conversion The solution: Scale image analysis on HPC Image analysis (at scale) Deliver results

13 Automated Image Analysis: Issues The solution: File conversion Scale image analysis on HPC Using bfconvert to go from.czi to.tif was crashing why? Even if it worked, it s in serial! Can we get HPC accelerated file conversion on GPFS? For now this is still I/O bound

14 Data mining GPFS: History Previously: SoNAS Home file system Big data file system NAS filesystem Currently: SoNAS + GRIDScaler SoNAS: Home file system DDN GRIDScaler (HPC)

15 Data mining GPFS: History SoNAS NFS exports to HPC, Desktops, etc. When HPC Heavy I/O workloads hit, all service was impacted HPC Desktops Etc. SoNAS

16 Data mining GPFS: History SoNAS NFS exports to HPC, Desktops, etc. When HPC Heavy I/O workloads hit, all service was impacted When SoNAS loads were high The cluster is broken Heavy I/O and CPU intensive work is affected LSF doesn t work /home doesn t work We need to throttle the queues, but can t! HPC SoNAS

17 Data mining GPFS: History Fix Integrate the NFS loads into the queuing system (via LSF elim ) and allow for dynamic dispatch control in response to external loads. HPC I/O load # of running jobs SoNAS

18 Data mining GPFS: History Fix Integrate the NFS loads into the queuing system (via LSF elim ) and allow for dynamic dispatch control in response to external loads. HPC Problem Dispatch control is cluster wide Everyone gets to dispatch or no one does CPU intensive work is still affected High NFS loads Stop dispatch Low NFS loads Allow dispatch SoNAS

19 Data mining GPFS: Today Using GPFS client tools we can see what each host is doing mmdiag: waiters mmpmon: Fine grained FS metrics Files opened (closed) per second MB read (written) per second Write (read) operations per second Clean / smooth the data GPFS Use filtered metrics for automated dispatch control and software analytics

20 Data mining GPFS: Today Waiters are noisy! From ~20 to ~100+ in 1 second

21 Data mining GPFS: Today Waiters are noisy! From ~20 to ~100+ in 1 second Perform a convolution to smooth out the data

22 Data mining your apps: Methods Use GPFScode (github) to mine GPFS loads,, along with the number of running jobs for each user,, and app,, has running at time Put it into one big matrix Expected value of due to would be:

23 Data mining your apps: Results AppA has high kwps but low MBWps. Buffer writes from AppA. This app was doing a gazillion writes ranging from bytes each. Throughput increased by ~9x Large numbers of small writes = Poor throughput

24 Data mining your apps: Results Efficiently caching data in GPFS pagepool increases reading substantially Measure Infiniband data movement Measure MBRps via GPFS At first, ib_in = MBRps, but then ib_in = 0 and MBRps skyrockets. This means we are caching the data locally and performing reads from memory. Currently looking at LROC

25 Questions?