DjiNN and Tonic: DNN as a Service and Its Implications for Future Warehouse Scale Computers

Transcription

1 and Its Implications for Future Warehouse Scale Computers Johann Hauswald, Yiping Kang, Michael A Laurenzano, Quan Chen, Cheng Li, Trevor Mudge, Ronald G Dreslinski, Jason Mars, Lingjia Tang University of Michigan Ann Arbor, MI

2 Intelligent Personal Assistant (IPA) Queries 2

3 Intelligent Personal Assistant (IPA) Queries Set my alarm for 6am 2

4 Intelligent Personal Assistant (IPA) Queries Set my alarm for 6am 2

5 Intelligent Personal Assistant (IPA) Queries What s the speed of light? Set my alarm for 6am 2

6 Intelligent Personal Assistant (IPA) Queries What s the speed of light? Set my alarm for 6am 2

7 Intelligent Personal Assistant (IPA) Queries What s the speed of light? Set my alarm for 6am Who is this? 2

8 Intelligent Personal Assistant (IPA) Queries What s the speed of light? Set my alarm for 6am Who is this? 2

9 Intelligent Personal Assistant (IPA) Queries What s the speed of light? Set my alarm for 6am Who is this? 2

10 3

11 Deep Neural Networks (DNNs) Feature Maps Inference 01 Spiderman 09 Superman 05 Batman Input Convolutional layer Pooling layer Fully Connected layer Network Architecture 4

12 Deep Neural Networks (DNNs) speech features Inference Who, is, this who is this word vectors w 0 w 1 w k w 0 w 1 w k w 0 w 1 w k Fully Connected layer Who (PRONOUN) is (VERB) this (PRONOUN) Network Architecture 4

13 DNN as a Service Image Classification Digit Recognition Facial Recognition Speech Recognition Natural Language Processing 5

14 DNN as a Service Image Classification Unified, highly optimized appliance for DNN Digit Recognition Facial Recognition Speech Recognition Natural Language Processing 5

15 We now have probably 30 or 40 different teams at Google using our infrastructure -Jeff Dean

16 DNN as a Service Image Classification Unified, highly optimized appliance for DNN Digit Recognition Facial Recognition Speech Recognition Natural Language Processing

17 Challenge Design a high throughput Warehouse Scale Computer (WSC) for DNN as a Service 8

18 Outline Identifying Bottlenecks for DNN as a Service Designing a High Throughput System Future Warehouse Scale Computer (WSC) Designs Conclusion 9

19 10

20 Tonic Suite Applications DjiNN Design Goals IMC Image Task DIG DjiNN DNN Service FACE Single web-service for DNN Diverse applications Speech Recognition (ASR) Low overhead request Task DNN Architecture processing It s business, Superman POS CHK NER IMC business (noun) Superman (P noun) It s (VP, B-NP) business (NP, I-NP) Superman (PERSON) DIG POS FACE CHK ASR NER Trained Models Natural Language Processing Task 11

21 Tonic Suite Applications DjiNN Implementation IMC Image Task DIG DjiNN DNN Service FACE DNN Architecture Decoupled architecture Arbitrary network architecture Speech Recognition (ASR) support Task Natural Language More details in paper Processing Task POS CHK NER business (noun) Superman (P noun) It s (VP, B-NP) business (NP, I-NP) Superman (PERSON) IMC DIG POS FACE CHK ASR NER Trained Models It s business, Memory resident models for Superman thread pool 12

22 Tonic Suite Tonic Suite Applications End-to-end applications that make requests to the DjiNN Service Span image, speech, and Users natural language processing State-of-the-art neural network architectures IMC Image Task DIG FACE Speech Recognition (ASR) Task It s business, Superman Natural Language Processing Task POS CHK NER business (noun) Superman (P noun) It s (VP, B-NP) business (NP, I-NP) Superman (PERSON) 13

23 Tonic Suite Applications IMC Users Image Task DIG DjiNN DNN Service FACE DNN Architecture Speech Recognition (ASR) Task It s business, Superman POS CHK NER IMC business (noun) Superman (P noun) It s (VP, B-NP) business (NP, I-NP) Superman (PERSON) DIG POS FACE CHK Trained Models Natural Language Processing Task ASR NER Release includes: inputs, pre-trained models, and modified Caffe djinnclarity-laborg 14

24 Identifying Bottlenecks for DNN as a Service 15

25 Identifying Bottlenecks for DjiNN and Tonic Software: Caffe (modified) CPU: Intel Xeon E GHz ATLAS (vectorized) DNN: More than 80% of cycles 16

26 Identifying Bottlenecks for DjiNN and Tonic Software: Caffe (modified) CPU: Intel Xeon E GHz ATLAS (vectorized) GPU: NVIDIA Tesla K40M cudnn v1 and Caffe 16

27 Identifying Bottlenecks for DjiNN and Tonic Software: Caffe (modified) 133x CPU: Intel Xeon E GHz ATLAS (vectorized) GPU: NVIDIA Tesla K40M ~7x cudnn v1 and Caffe 16

28 Identifying Bottlenecks for DjiNN and Tonic Throughput Improvement GPU Profiling 16

29 Identifying Bottlenecks for DjiNN and Tonic Low memory bandwidth utilization Throughput Improvement GPU Profiling 16

30 Identifying Bottlenecks for DjiNN and Tonic Throughput Improvement GPU Profiling GPU is not fully utilized 16

31 Designing a High Throughput System 17

32 Designing a High Throughput System Batching x0 x01 x02 x03 x04x05 Batch Size: x1 x11 x12 x13 x14x15 xd xd xd xd xd xd Streaming Multiprocessor WARP WARP WARP WARP WARP WARP High Occupancy 18

33 Designing a High Throughput System Batching x0 x01 x02 x03 x04x05 Batch Size: x1 xd x11 xd x12 xd x13 xd x14x15 xd xd ~15x Streaming Multiprocessor ~5x WARP WARP WARP WARP WARP WARP High Occupancy 18

34 Designing a High Throughput System Concurrent Execution Launch concurrent DNN services on the GPU Leverage NVIDIA Multi- Process Service (MPS) [1] SM0 GPU SM1 SM14 [1] Multi-Process Service Multi_Process_Service_Overviewpdf 19

35 Designing a High Throughput System Concurrent Execution Launch concurrent DNN services on the GPU Leverage NVIDIA Multi- Process Service (MPS) [1] 4 DNN concurrent instances Average improvement: 3x SM0 GPU SM1 SM14 [1] Multi-Process Service Multi_Process_Service_Overviewpdf 19

36 Designing a High Throughput System ~127x ~7x Average throughput improvement: 133x 20

37 1 GPU

38 8 GPU

39 GPU Scaling Average throughput improvement using optimizations: 771x 22

40 Bandwidth Requirements for Peak Throughput Experimental setup: eliminate any data transfer to GPU Bandwidth to GPUs saturated NLP requires more bandwidth to GPUs 23

41 Single Server Key Insights DNNs do not benefit equally from optimizations High communication Natural Language Processing (NLP) tasks require far more bandwidth Optimizing compute platforms depends on DNN s computation and communication characteristics 24

42 Future Warehouse Scale Computer (WSC) Designs 25

43 Future WSC Designs Server Designs 26

44 Future WSC Designs Server Designs 26

45 Future WSC Designs Server Designs CPU-only: +no extra HW -low throughput 26

46 Future WSC Designs Server Designs CPU-only: +no extra HW -low throughput 26

47 Future WSC Designs Server Designs CPU-only: +no extra HW -low throughput 26

48 Future WSC Designs Server Designs CPU-only: Integrated GPU: +no extra HW +homogeneous -low throughput -over-provision GPU 26

49 Future WSC Designs Server Designs CPU-only: Integrated GPU: +no extra HW +homogeneous -low throughput -over-provision GPU 26

50 Future WSC Designs Server Designs CPU-only: Integrated GPU: +no extra HW +homogeneous -low throughput -over-provision GPU 26

51 Future WSC Designs Server Designs CPU-only: Integrated GPU: +no extra HW +homogeneous -low throughput -over-provision GPU Disaggregated GPU: +decouple CPU/GPU -data transfer 26

52 Future WSC Designs Total Cost of Ownership (TCO) Expand Barroso [1] model with GPU and networking costs Addressing the bandwidth bottleneck further improves NLP TCO (more details in paper) Integrated GPU Disaggregated GPU TCO (Normalized to CPU-only) x MIXED IMAGE NLP 5x [1] Barroso, Luiz André, et al "The datacenter as a computer: An introduction to the design of warehouse-scale machines" 27

53 WSC Scale Key Insights TCO improvement dependent on the DNN workload composition Bandwidth constrained workloads underutilize GPUs in Integrated design Sufficient bandwidth is critical to fully utilize resources 28

54 Conclusion Unified, state-of-the-art, highly optimized DNN as a Service DNNs have different compute and communication characteristics Do not benefit equally from optimizations Characteristics impact system designs 29

55 Thank you djinnclarity-laborg 30