Dive into nova scheduler performance

Transcription

1 Dive into nova scheduler performance Where is the bottleneck? Cheng, Yingxin

2 Contents Background Performance, Optimizations (58 simulated runs) Future

3 Background Scheduler architecture Analysis methodology Experiment assumptions

4 Scheduler architecture Pre-scheduling Scheduling Post-scheduling

5 Analysis methodology Logs Faked Knowledge

6 Experiment assumptions Controller and compute are in separate machines Environment Newton RC2( rc2) 48 threads: 2 x Intel Xeon Processor E v3 (30M Cache, 2.50 GHz) 128 GB RAM: 8 x 16GB HP DDR4 3 TB Disk Faked Post scheduling, network interfaces *Other names and brands may be claimed as the property of others. 6

7 Performance One scheduler deployment range(200, 1001, 200) Simulated nodes [1, 25, 50, 75, 100] Stressed requests

8 Nodes (1 st experiment) Unstressed performance [200, 400, 600, 800, 1000] nodes 1 request 1 scheduler Schedule overhead: 200 nodes: 29.1% 1000 nodes: 64.0% 8

9 Nodes (1 st experiment) Unstressed scheduling SchedulerManager.select_destinations() FilterScheduler._schedule() FilterScheduler._get_all_host_states() Cache-refresh overhead: 91.1 ± 1.1% Cache-refresh is blocking the entire schedule process 9

10 Nodes (2 nd experiment) Stressed performance [200, 400, 600, 800, 1000] nodes 50 concurrent requests 1 scheduler Schedule overhead (with wait): 200 nodes: 72.3% (29.1%) 1000 nodes: 91.6% (64.0%) 10

11 Nodes (2 nd experiment) Stressed scheduling Even larger cache-refresh bottleneck: 98.5 ± 0.2% (91.1 ± 1.1%) The performance metric: throughput 4.33~1.08 query per second 200~1000 nodes 1 scheduler 50 requests Do requests changes performance? 11

12 How concurrency changes performance? (3 rd : 1~200 requests, 400 nodes, 1 scheduler) Saturated scheduling 12

13 How concurrency changes scheduling? (3 rd : 1~200 requests, 400 nodes, 1 scheduler) Saturated scheduling 13

14 How concurrency changes throughput? (3 rd : 1~200 requests, 400 nodes, 1 scheduler) Performance of this 400 node simulation is ~2.7 queries/second 14

15 The performance of nova scheduler Messages REQ API Conductor Scheduler Conductor Compute VM Nova scheduler can be saturated by requests. Schedule requests can be piled up before scheduler service. Schedule overhead: 72.3~91.6% (200~1000 nodes, 50 requests) Cache-refresh overhead: 98.5% (200~1000 nodes, 50~200 requests) The entire scheduling is blocked by cache-refresh from database. Performance: 4.3~1.1 requests per second (200~1000 nodes) 15

16 optimize More schedulers? More workers? Caching scheduler?

17 More schedulers? [1, 2, 4, 8, 16] schedulers 400 nodes, 50 requests Schedule overhead (with wait): 80.5~17.3% (1~16 schedulers) Query time saving: up to 64.9% (8 schedulers) Query time increased? 17

18 More schedulers Pros and cons Pros Moderated bottleneck Query time saving: up to 64.9% Throughput improvement: up to 337% Cons Scheduler decision collision Placement accuracy Performance overhead 18

19 More workers? [1, 2, 8, 12, 16] api/conductor workers 1 scheduler, 400 nodes, 50 requests Configuration [DEFAULT] osapi_compute_workers [conductor] workers Performance improvement: Mainly in API service Not significant in scheduling 19

20 Caching scheduler? (200~1000 nodes, 1 scheduler, 50 requests) 64.9~87.7% time saving 20

21 Caching scheduler Pros and cons Configuration [DEFAULT] scheduler_driver = caching_scheduler Pros: Query time saving: 64.9~87.7% Throughput improvement: 342~850% Cons: Outdated resource view Placement accuracy Multiple scheduler support 21

22 Schedule visualization 22

23 Optimization strategies Database: difficult but effective Caching scheduler Up to 512% improvement in throughput (400 nodes) Outdated resource view Race conditions: % (2 schedulers, 30 nodes, 315 requests) More filter schedulers Up to 337% improvement in throughput (400 nodes) Race conditions: % (2 schedulers, 30 nodes, 315 requests) More workers: tiny improvements 23

24 future Utopia scheduling Upcoming schedulers

25 Utopia scheduling Scheduling Improve database: Memory-based DB Optimize SQL: Resource-provider scheduler Exclude database: Shared-state scheduler Pre-scheduling Nova-api optimization Post-scheduling Nova-compute optimization Split real-time request processing out of one big thread 25

26 Upcoming schedulers 26

27 Epilog

28 Further readings Nova scheduler benchmarking data One scheduler racing problem Shared-state scheduler Resource-provider scheduler 28

29 Try the tool in your environment Ask No test environment? Try OSIC-Cluster: Nova Scheduler Dev Meeting Weekly on Monday at 1400 UTC in #openstack-meeting-alt 29

30 Acknowledgement Jing, Yuntong (Intel) Zhou, Zhenzan (EMC) Liu, Junwei (China Mobile Suzhou Research Center) 30

31 Performance Nodes, requests, throughput Optimize More schedulers More workers Caching scheduler Future Utopia scheduling Upcoming schedulers 31

32 Legal notices and disclaimers Intel technologies features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit Intel, the Intel logo and others are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others Intel Corporation.

33