I/O Performance and I/O Performance Isolation

Transcription

1 I/O Performance and I/O Performance Isolation Sarala Arunagiri Representing the DAiSES Project at The University of Texas at El Paso 17 April 2007 HiPCAT Meeting at UTEP 1

2 Outline of the Talk What is I/O Performance? Storage I/O Problem Illustrative Example MADbench Fairness and Performance Isolation 17 April 2007 HiPCAT Meeting at UTEP 2

3 What is I/O Performance? Performance means different things to different people. Performance concerns of resource manager Efficient resource utilization which translates to maximizing the throughput for a given set of I/O requests Performance perceived by users Commonly users perceive response time Resource utilization and response times could be uncorrelated 17 April 2007 HiPCAT Meeting at UTEP 3

4 I/O Bottleneck Therefore, for I/O intensive jobs, I/O Performance tuning is crucial 17 April 2007 HiPCAT Meeting at UTEP 4

5 How far away is your data? The actual latency for Internal Transfer is of the order of nanoseconds and the latency for Disk devices is of the order of milliseconds Illustration of Relative Latencies Source: Tom West, hyperi/o LLC., 'How fast are your files? A Case for Monitoring File I./O Performance', July 2001 (This figure is an adaptation of the How far away is your data? figure shown in Jim Gray, "Computer Technology Forecast for Virtual Observatories", Microsoft Technical Report MSR- TR , September 2000). 17 April 2007 HiPCAT Meeting at UTEP 5

6 Storage I/O Problem 17 April 2007 HiPCAT Meeting at UTEP 6

7 I/O Performance tuning for MADbench MADbench is a lightweight version of the MADCAP Cosmic Microwave Background (CMB) code It is written using MPI and has 3 distinct I/O phases: dsdc: write matrices to disk invd: read matrices from disk W: read matrices from disk MPI tasks can thus be writers and readers 17 April 2007 HiPCAT Meeting at UTEP 7

8 Cause of Multiple I/O Streams Generally File I/O is parallelized along with computation in HPC applications 17 April 2007 HiPCAT Meeting at UTEP 8

9 Linux CFQ Scheduler 17 April 2007 HiPCAT Meeting at UTEP 9

10 File layout Single writer Intel Xeon with four logical processors, one I/O node, and EIDE disk. Blktrace used for analysis of seek behaviour 17 April 2007 HiPCAT Meeting at UTEP 10

11 MADbench Read Behavior Four readers Resource utilization is high! Although it is not efficient. 17 April 2007 HiPCAT Meeting at UTEP 11

12 Anticipatory Scheduler Resource utilization decreases since wait times are introduced 17 April 2007 HiPCAT Meeting at UTEP 12

13 Illustration of relative times involved in a Magnetic Disk Access 17 April 2007 HiPCAT Meeting at UTEP 13

14 Anticipatory Scheduler Resource utilization decreases since wait times are introduced 17 April 2007 HiPCAT Meeting at UTEP 14

15 Execution time of MADbench CFQ AS Execution time with CFQ is more than that with AS 17 April 2007 HiPCAT Meeting at UTEP 15

16 Seek Behavior Nice and sequential with an occasional seek Lots of seeking taking place! Seeking increases with number of tasks 17 April 2007 HiPCAT Meeting at UTEP 16

17 The next question Anticipatory Scheduler seems to give rise to improved performance as comapred to CFQ. However, is that the best we can do in terms of I/O Performance? 17 April 2007 HiPCAT Meeting at UTEP 17

18 Single Reader 17 April 2007 HiPCAT Meeting at UTEP 18

19 Execution time CFQ AS 17 April 2007 HiPCAT Meeting at UTEP 19

20 Multiple Writers in MADbench 17 April 2007 HiPCAT Meeting at UTEP 20

21 Intel Xeon 1GB App Throttling 17 April 2007 HiPCAT Meeting at UTEP 21

22 Is one-reader and one-writer a good rule of thumb for all cases? 17 April 2007 HiPCAT Meeting at UTEP 22

23 One-writer and varying number ofreaders p690: 2 or 4 readers is the sweet spot Cluster: 8 readers is the sweet spot 17 April 2007 HiPCAT Meeting at UTEP 23

24 Conclusions drawn from MADbench experiments The data collected so far indicates that several factors along the I/O stack influence the optimal number of readers and the optimal number of writers. We shall continue to conduct relevant experiments and analyze data required to obtain sufficient insights for dynamic adaption of the tuning parameters. 17 April 2007 HiPCAT Meeting at UTEP 24

25 Determinism in Performance Any determinism that is introduced into the I/O performance e.g., response time guarantees, fairness, performance isolation, will be satisfied by paying a price in terms of a possibly reduced throughput. 17 April 2007 HiPCAT Meeting at UTEP 25

26 Multiple Contending Applications Performance concerns of resource manager Efficient resource utilization Users would like Good performance Fairness in resource sharing and Performance Isolation.. Let us look at the simplest of scenarios; applications of a system contending for a single disk 17 April 2007 HiPCAT Meeting at UTEP 26

27 Can Linux CFQ scheduler provide Performance Isolation? 17 April 2007 HiPCAT Meeting at UTEP 27

28 DAiSES Research A PhD Thesis and a Master's Thesis; Five publications and a patent Work includes Mathematical Analysis of different ways of measuring fairness. A Scheduling algorithm CFQ-CRR that provides fair sharing of 'storage service time' among applications. In addition, it facilitates the the usage of an I/O scheduling algorithm of its choice, by the application, thus enabling dynamic adaptivity. Three new scheduling algorithms co-operative anticipatory scheduler Complete Fair Queuing with Compensated Round Robin (CFQ-CRR) CFQ-CRR(p) 17 April 2007 HiPCAT Meeting at UTEP 28

29 CFQ-CRR: Performance Isolation when request sizes vary CFQ-CRR: Execution Times of Different Instances of Applications 1 and 2. Application 1 always has a Fixed Request Size, while the Request Size of Application 2 Varies 17 April 2007 HiPCAT Meeting at UTEP 29

30 CFQ-CRR: Performance Isolation in the face of varying seek characteristics VIOS Execution Times of Different Instances of Programs 1 and 2. Program 1 always has a Fixed Inter-request Seek Distance, while that of Program 2 Varies 17 April 2007 HiPCAT Meeting at UTEP 30

31 Tradeoff Execution time Vs Performance Isolation Execution Time (sec) Row 34 Row 32 Execution Time (sec) Row 34 Row CFQ- CRR(P) CFQ- CRR CFQ- Linux Noop Application execution times of the threads that finished first and last among 32 concurrent threads; 1000 random 4KB requests/threads 0 Deadline Anticipatory CFQ- CRR(P) CFQ- CRR CFQ- Linux Deadline Anticipatory Noop Maximum and average latency of requests with different schedulers; each thread accesses disjoint areas of the disk 17 April 2007 HiPCAT Meeting at UTEP 31

32 Importance of Performance Isolation When provided by a storage service provider, it is a desirable QoS guarantee. In a Multiprocessing environment, it increases productivity by easing debugging effort, in some cases. Aids in locating load imbalances. 17 April 2007 HiPCAT Meeting at UTEP 32

33 Survey of Performance Isolation in Modern Scheduling Algorihtms Schedulers/tools Cello disk scheduling framework Yes Absolute Performance Isolation CFQ-CRR(P) YFQ Facade virtual store controller Interposed 2-Level scheduler Triage workload controller, both file and block access SLEDS-storage controller, block level storage service Chameleon-storage resource arbitrator Yes No No No No Policy can be set to achieve the goal. Heuristic presented does not provide Absolute Performance Isolation. Policy can be set to achieve the goal. 17 April 2007 HiPCAT Meeting at UTEP 33

34 Work In Progress at DAiSES Major Areas of I/O Research;; Working towards dynamic adaptivity for I/O Performance in HPC applications such as MADCAP and WRF codes Designing I/O Scheduling Algorithm which works for a RAID Storage System, with following features; -Provides a good throughput -Services requests according to the latency requirements of different applications -Provides performance isoaltion among the contending applications Exploring ways to handle the enormous I/O requirements of checkpointing applications. Thank You. 17 April 2007 HiPCAT Meeting at UTEP 34