RE-EVALUATING RESERVATION POLICIES FOR BACKFILL SCHEDULING ON PARALLEL SYSTEMS

Transcription

1 The th IASTED Int l Conf. on Parallel and Distributed Computing and Systems (PDCS), Cambridge, MA, Nov. RE-EVALUATING RESERVATION POLICIES FOR BACKFILL SCHEDULING ON PARALLEL SYSTEMS Su-Hui Chiang Computer Science Department Portland State University Portland, Oregon suhui@cs.pdx.edu Chuyong Fu Computer Science Department Portland State University Portland, Oregon chuyf@cs.pdx.edu ABSTRACT On parallel systems, jobs that request a large fraction of the maximum resources available on the system may incur poor wait time. This paper evaluates whether giving a reservation to every waiting job can improve large jobs without significantly degrading the performance of other jobs. Using a wide range of workloads, including more recent workloads than SP workloads, and a more complete set of performance measures than in previous studies, we provide new observations of potential benefit and problem of reservation policies that give all jobs a reservation. KEY WORDS Scheduling, backfill, reservation, workload analysis. Introduction Nonpreemptive job scheduling are used on many production systems, due to the advantage of a low scheduling overhead compared to preemptive policies. Job scheduling on such systems are particularly challenging in the presence of large-resource jobs, each of which requires a large fraction of the resource in the system, because it is difficult to find enough resources for such jobs on highly utilized systems and to balance the scheduling of such large jobs against responsive scheduling of a very large number of smaller jobs with highly variable runtimes. Reserving resources for a waiting job to start in the future, as in backfill policies, has the potential to help large jobs. Three versions of reservation policies have been studied for FCFS-backfill previously. One version gives only one job a reservation; the other two versions give all waiting jobs a reservation but differ in how they update scheduled start times when a job terminates earlier than expected. due to inaccurate s. Many papers evaluated these reservation policies. The results suggested that workloads have an impact on relative policy performance. However, several questions remain to be studied. First, the data in [,, ] show that using only one reservation achieves better average response time and slowdown than that of a particular version of giving all waiting jobs a reservation under several SP workloads, but a different result was found for a particular synthetic workload []. The authors in [] attributed the different results to the differences of the workloads. It remains to be studied whether their hypothesis holds for real traces from non-sp systems. Second, the paper [3] compared the two versions of FCFS-backfill that give all jobs a reservation, but they used only a particular SP workload. It is not clear whether their results hold for other workloads. Finally, previous studies have focused on average performance measures and overlooked potential performance problem for large jobs. In this paper, we re-evaluate the three reservation policies for FCFS-backfill using recent monthly workloads on the Intel-IA cluster from NCSA as well as previously used SP workloads. The key contribution of this paper includes: () compare recent workloads on an IA cluster at NCSA and three previous SP workloads; () show quantitative performance difference between the largest jobs and smaller jobs; (3) provide more complete evaluation of the three reservation policies for FCFS-backfill, complementing previous results, while offering new observations. The remainder of this paper is organized as follows. Section defines the reservation policies with a brief review of previous results. Section 3 compares the workloads used. Section presents the results for policy evaluation. Section summarizes our results. Background This section provides the definition of the three versions of FCFS-backfill and a brief review of related work. For convenience, we use the notation defined in Table. Table summarizes the key features of three versions of FCFS-backfill policies. Under FCFS-backfill, jobs are prioritized in the order of arrival time. The FCFS-backfill/r= policy reserves resources for the first waiting job to start at the earliest time when enough resources will be available. Jobs from the back of the queue can start on the currently free resources as long as they Symbol N R T Table. Notation Definition Number of User in the traces Actual job runtime

2 Table. Key Features of Three Versions of FCFS-backfill FCFS-backfill Versions Feature FCFS-backfill/ r= (also known as EASY) Only the first job receives a reservation (i.e., scheduled start time). FCFS-backfill/ r=all/ compress Every waiting job receives a reservation; (also known as Conservative) on a job departure, compress scheduled start times to preserve their order. FCFS-backfill/ r=all/ renew Every waiting job receives a reservation; (another Conservative version) on a job departure, renew scheduled start times in the order of job arrivals. Table 3. Workload Comparisons: System Capacity and Job Size Job Size System Capacity Limit Average Workload (Max. # Nodes) Requested Actual Period N R N R T NCSA-IA h 9. 3.h.7h May 3 - Nov. 3 (Titan) h.9.h 3.h Dec. 3 - Apr. CTC-SP 3 33 h.7.h 3.h July 99 - May 997 KTH-SP h h.h Oct Aug. 997 SDSC-SP h..h.7h May 99 - Apr. will not delay the scheduled start time of the first waiting job. This policy was called EASY in previous papers (e.g., [, ]). Both FCFS-backfill/r=all/renew and FCFSbackfill/r=all/compress give a reservation to all waiting jobs. They are called Conservative in many previous papers. As explained in the table, they differ in how they update scheduled start times when a job departs earlier than expected. Mu alem and Feitelson [] provide an example illustrating the difference. They argued in favor of r=all/compress against r=all/renew because r=all/compress provides an estimated start time for each job and guarantees that the job will not start later than that. However, we will show later that these start time estimates can be very inaccurate because they are computed from potentially very inaccurate s. Several papers compared one or two versions of FCFS-backfill. Perkovic and Keleher [3] compared r=all/compress and r=all/renew (called with guarantee and no guarantee, respectively). They showed that FCFS-backfill with r=all/compress achieves a lower average wait time and similar average slowdown under a CTC SP workload. Several papers [,, ] compared r= and r=all/compress and found that r= has similar or significantly lower average response than that of r=all/compress for SP workloads. However, the results in [] also showed that r=all/compress has a better average response than that of r= for a synthetic workload. The authors of the paper attribute the different results to the difference in the workloads, i.e., jobs in the synthetic workloads have a larger average number of and shorter runtime than in SP workloads. Srinivasan et al. [] further showed that r= favors jobs that are long and narrow but r=all/compress favors jobs that are short and wide under extremely heavy load workloads constructed from SP traces. Finally, Chiang et al. [] studied the impact of the number of reservations (renew version) on backfill policies, including FCFS-backfill. They showed that using a few reservations (-) improves the maximum wait without sacrificing average performance measures, but more than a few reservations can be detrimental for the workloads studied. In addition, more accurate job s can significantly improve backfill policies, compared to using user estimates. Similar results were shown in [, ]. 3 Workload Comparisons This section summarizes and compares workloads on four systems: the NCSA-IA cluster (also known as Titan) and three previous SP systems from CTC, KTH, and SDSC. Table 3 summarizes the system capacity, the maximum and average job size, and the duration of each trace. The -month Titan trace from the NCSA ran between May 3 and April. We divide this trace into two periods by Dec. 3, during which the limit was increased from to hours. The three SP workloads were used in many previous papers, with the duration shown in the table. Interesting features include: () the largest job in each system requires a large fraction (7% - %) of the maximum resource available on the system; () as the limit doubled on Titan, the average requested and actual runtime also doubled, but the average number of job decreased by about %; (3) Titan used a similar or smaller limit on the requested nodes than on SP systems, but Titan workloads have a larger number of in average. For Titan workloads, Table further shows the average,, and actual runtime of each job for each month. Each month is labeled (e.g.,

3 Table. Average Job Size on Monthly Titan Workloads Month Month N R T (R h) (R h) N R T May 3 (L n ) 9. 3.h.h Dec 3 (S n M t )..h 3.h Jun 3 ( )..h.h Jan ( L t ).7.h.h Jul 3 (L n ) 3..h.9h Feb (S n M t )..h 3.h Aug 3 ( ) 7.3.h.h Mar (S n )..h.h Table. Classification of Titan Monthly Workloads Requested Nodes Runtime S n L n M t L t Prob(N ).7 Prob(N ). (typically N ) & N N > R < h R : - 7h R > 7h L t ) based on the average and runtime of the month. The letters S, M, L stand for small, medium, and large, defined in Table. Note Sep. - Nov. 3, omitted to conserve space, are or S n. For simplicity of graphs in Section, we show results for four months, marked by a in the table. These months provide a variation of workloads. The table shows that average job sizes vary across months. In particular, () May and July 3 have a much larger average number of (9-3) than in other months; () the average is - hours during May - Nov. 3, and increases to - hours during Dec. 3 - April. Note that June ( ) and July 3 (L n ) may be in favor of r=all policies because of a relatively large average and short runtime, based on results in [, ]. Next, we show the workload distributions for four representative workloads: two SP workloads (CTC and KTH) and two Titan monthly workloads (7/3 and 3/). Figures -3 plot respectively the distribution of requested nodes,, and (shown as average and -, -, and -percentile) for each range of in each workload. As shown in Figures -, the Titan 7/3 workload (L n ) is most similar to the synthetic workload used in [] in that the requesting one node is significantly small (% smaller) and that requesting a short runtime ( hour) is large (% larger), compared to the CTC workload. Figure 3(a)-(d) show that the relation between and number of nodes varies across workloads. In both CTC and Titan 7/3 workloads (Figure 3(a) and (c)), jobs with the smallest and largest tend to request a longer runtime; in the KTH workload (Figure 3(b)), jobs in the middle range of tend to request a longer runtime; in the Titan 3/ (S n ) workload (Figure 3(d)), jobs requesting one node tend to request a much shorter runtime than that of larger jobs. Some comments on other workloads not shown: the SDSC workload has the strongest positive correlation between R and P; most Titan months are more similar to Figure 3(a) and (c), while others are more similar to (d); only one Titan month (/3) has some similarity to (b). Performance Evaluation In this section, we evaluate the three versions of FCFSbackfill policies, defined in Table. Section. presents results of using user-estimated job runtimes for scheduling and. the results of using actual job runtimes. We evaluate policies by an event-driven simulator, using Titan monthly workloads and SP workloads, discussed in Section 3. To be realistic, each simulation includes a warm-up and a cool-down period, using jobs submitted prior to and after the period for which performance is measured. Only jobs submitted during the period for performance measurement are analyzed. To compare performance across different months at the same load, we shrink or increase job interarrival times to create a target load level, as in previous studies [, ]. Three levels of offered load, ρ, for each workload are evaluated:.,., and.9, computed as the processor time (i.e., P T) of jobs submitted in the period for performance measurement, divided by the maximum processor time available in that period. Note that ρ =.9 is high but can be reasonably expected. For example, the heaviest-load month in Titan (/) trace is.9. The results are shown for ρ =. and.9. The results for. are qualitatively similar to that of.. For simplicity, only four Titan workloads and two SP workloads are shown. Other workloads will be commented.. Comparisons of Reservation Policies In this section, we compare the three versions of FCFSbackfill, i.e., r=, r=all/renew, and r=all/compress, using requested job runtimes given by the users for scheduling. We also evaluated FCFS-backfill with a few reservations (up to ), omitted for simplicity; they are more or less similar to that of FCFS-backfill/r= (i.e., one reservation). Figure compares policies under the three SP workloads. Figure (a) plots the average wait of jobs in each policy for ρ =.; Figure (b)-(d) plot respectively the average wait, maximum wait, and average slowdown of jobs for ρ =.9. Similarly, Figure plots the results for representative Titan monthly workloads. The key results are 3

4 jobs requesting power of processors other jobs (a) CTC (b) KTH (c) Titan 7/3 (L n ) (d) Titan 3/ (S n ) Figure. Example Distributions of Requested Nodes jobs requesting upper bound of range other jobs m h h h h h h h (a) CTC m h h h h h h h h (b) KTH m h h h h h h h (c) Titan 7/3 (L n ) m h h h h h h h (d) Titan 3/ (S n ) Figure. Example Distributions of Requested Runtime (hour) 3 (a) CTC (hour) percentile avg percentile percentile 3 (b) KTH (hour) 3 (c) Titan 7/3 (L n ) Figure 3. Example Distributions of Requested Runtime for Each Range of Requested Nodes (hour) 3 (d) Titan 3/ (S n ) summarized below. First, consistent with results in [, ], r= has better or similar performance compared to that of r=all policies for SP workloads as well as Titan monthly workloads. The only exception is the Titan March workload, shown in Figure (d), in which r= has twice as large average slowdown as that of both r=all policies, because r=all favor short jobs, to be shown later. Results for other Titan monthly workloads are similar to the other three Titan workloads shown. Note also that contrary to the hypothesis in [] (discussed in Section -3), we do not find that r=all outperform r= for Titan 7/3 or /3 workloads (except a small difference for 7/3 with ρ =.9). Second, r=all/compress has a potential problem of very poor maximum wait time. In particular, the maximum wait of r=all/compress almost doubles that of r=all/renew and r= for the SDSC workload (Figure (c)) and Titan 3/ (Figure (c)). In addition, both r=all policies have similar average wait time in most cases, except for () the CTC workload with ρ =. (Figure (a)), in which r=all/compress has 3% lower average wait than that of r=/renew, consistent with results in [3]; () the SDSC workload with ρ =.9 (Figure (b)), in which r=all/renew has % lower average wait than that of r=all/compress. Finally, the offered load has some impact on relative policy performance. In particular, the average wait time of r=all/compress relative to that of r=all/renew for the Titan monthly workloads slightly improves as the load increases from. to.9, as shown in Figure (a)-(b). In addition, the average slowdown of r= relative to that of the r=all policies for the Titan workloads degrades as the load increases to.9. However, the change in most cases is small, except for the average slowdown in 3/ (Figure (d)). Next, we further evaluate the performance of different job classes under each policy, for two representative workloads: KTH-SP and Titan 3/. Figure (a) and (b) plot the average wait for each range of under each workload. The figure shows that both r=all policies significantly penalize long s, compared to r=; in the case of the Titan 3/ workload (graph (b)), both policies greatly improves s up to hours. The 3/ results explain the exceptionally lower average slowdown under r=all than under r= in that month. Note that in most other workloads studied, r=all favor short jobs, but the range of s improved and the degree of improvement are much smaller than for 3/. Figure 7(a)-(b) plot the results for each range of requested nodes. First, notice that large jobs incur much worse wait than of smaller jobs. Compared to r=, using r=all/renew slightly improves large jobs but at the great expense of small jobs for the KTH workload; on the other hand, r=all/renew does not improve but in fact have worse

5 (a) Avg. Wait ρ =. (b) Avg. Wait ρ =.9 max wait time (hour) FCFS bf/r= (c) Max. Wait ρ =.9 Figure. Previous SP Workloads: Comparisons of Three Versions of FCFS-backfill avg slowdown (d) Avg. Slowdown ρ =.9 /3 7/3 / 3/ L n L t S n (a) Avg. Wait ρ =. /3 7/3 / 3/ L n L t S n (b) Avg. Wait ρ =.9 max wait time (hour) 3 FCFS bf/r= /3 7/3 / 3/ (c) Max. Wait ρ =.9 Figure. New Titan Workloads: Comparisons of Three Versions of FCFS-backfill avg slowdown /3 7/3 / 3/ (d) Avg. Slowdown ρ =.9 3 m h h h h h h h h (a) KTH-SP 3 FCFS bf/r= m h h h h h h (b) Titan 3/ (S n ) Figure. Performance versus Requested Runtime (ρ =.9) (a) KTH-SP FCFS bf/r= (b) Titan 3/ (S n ) Figure 7. Performance versus Requested Nodes (ρ =.9) performance for large jobs for Titan 3/ workload. Results for most other workloads studied are more similar to the 3/ results. The different results should be due to workload correlation. Recall from Figure 3, the largest and smallest jobs in KTH have a similar distribution of requested runtime; while in Titan 3/, jobs requesting more than one node tend to request a much longer runtime than that of one-node jobs. With many short one-node jobs in 3/ (Figure (d)), it is difficult for large and long jobs to compete. Figure 7 also shows r=all/compress has very poor performance for large jobs while not improving small jobs. Results for almost all other workloads are similar. Our key conclusions are () r=all may favor short jobs (similar to []) but in most cases do not favor large jobs; in fact, jobs that are both large and long may suffer much more under r=all than under r=; () correlation between and number of nodes is as important a factor as the distributions of these measures, on relative policy performance. Finally, we show that for jobs that have waited in the queue, there is a large error in the estimated wait time provided by r=all/compress. Table shows the best, average, and worst cases, of the errors in the estimated wait time for Titan workloads. For each case, the table shows the average of the estimated and actual job wait time, and the average, median, and maximum error. Even in the best case, the average error is more than the average actual wait, while the worst case has an average error of close to hours!. Impact of Using Actual Job Runtimes We showed in [] that using actual job runtimes can significantly improve priority backfill policies using a few reservations. In this section, we study to what extent can using actual runtimes improves r=all policies. Note that if actual runtimes are used, using r=all/renew and r=all/compress for FCFS-backfill are equivalent. Figure (a)-(c) plot respectively the average wait, maximum wait, and average slowdown, under FCFSbackfill with r= and r=all in each of the four Titan monthly

6 Table. FCFS-backfill/r=all/compress: Errors in Initial Estimated Start Time Workload Average Wait Wait > Estimation Errors (W est - W act ) (Titan; ρ =.) Estimate (W est ) Actual (W act ) Average Median Maximum Jun 3 (best case).h.h.h.9h 7.9h Aug 3 (avg. case).3h 3.h 7.h.h 9.h Jan (worst case) 9.h.h 9.h.h.h /3 7/3 / 3/ (a) Avg. Wait max wait time (hour) FCFS bf/r= (R*=R) FCFS bf/r= (R*=T) FCFS bf/r=all (R*=T) /3 7/3 / 3/ (b) Max. Wait avg slowdown /3 7/3 / 3/ L n L t S n (c) Avg. Slowdown Figure. Potential Benefit of Using Actual Job Runtimes (ρ =.9) (R*: Simulated ) (d) W vs. N (Representative, /) workloads. For r=, the results of both using user-estimated runtimes (i.e., R* = R) and using actual runtimes (i.e., R* = T) are shown. For r=all, only results of R* = T are shown for simplicity. As expected, using actual runtimes considerably improves r=. The benefit is even larger for r=all, as r=all now performs similarly to that of r=, implying that inaccurate s have caused performance problem of r=all. Figure (d) plots the average wait for each range of requested nodes under each policy for a representative workload. The graph shows that using actual runtimes for r= considerably improves relatively large jobs (N > ) without hurting smaller jobs. In some months (e.g., Titan /, not shown), all ranges of are considerably improved. The results for the maximum wait are qualitatively similar, not shown to conserve space. Conclusions In this paper, we re-evaluated the performance of different reservation policies for FCFS-backfill. A wide range of workloads were used, including recent monthly workloads from NCSA Intel-IA (i.e., Titan) over one-year period and three previous SP workloads. We identify potential problems of the reservations policies by using a more complete set of policy performance measures. Our results also include a comparison of the workloads used. Below summarizes key results. First, giving all waiting jobs a reservation and compressed rescheduling (i.e., r=all/compress) has a potential problem of very poor performance for large and/or long jobs, compared to using one reservation (i.e., r=). Second, with renew rescheduling, r=all/renew can alleviate the problem of r=all/compress by allowing rescheduling of jobs that are large and/or long to start in front of smaller jobs that arrive later. However, in comparison, r= is simpler in implementation and has similar or better overall performance for a wide range of workloads studied. Third, the benefit of using actual runtimes is greater if all jobs are given a reservation (i.e., r=all) than for r=, resulting in more similar performance for r= and r=all. However, since majority of job s are potentially very inaccurate for expected workloads, using one (or a few) reservation may be preferred. Our on-going work includes studying alternative approaches to improve large jobs in the workloads. References [] S.-H. Chiang, A. Dusseau-Arpaci, and M. K. Vernon. The impact of more accurate s on production job scheduling performance, Proc. th Workshop on Job Scheduling Strategies for Parallel Processing, Edinburgh, Scotland, July. [] A. W. Mu alem and D. G. Feitelson. Utilization, predictability, workloads, and user runtime estimates in scheduling the IBM SP with backfilling, IEEE Trans. Parallel and Distributed Syst., ():9 3, June. [3] D. Perkovic and P. J. Keleher. Randomization, speculation, and adaptation in batch schedulers, Proc. ACM/IEEE Supercomputing Conf., Dallas, Nov.. [] W. Smith, V. Taylor, and I. Foster. Using run-time predictions to estimate queue wait times and improve scheduler performance, Proc. th Workshop on Job Scheduling Strategies for Parallel Processing, San Juan, April 999. [] S. Srinivasan, R. Kettimuthu, V. Subramani, and P. Sadayappan. Characterization of backfilling strategies for parallel job scheduling, Proc. ICPP Workshop on Scheduling and Resource Management for Cluster Computing,. [] S. Srinivasan, R. Kettimuthu, V. Subramani, and P. Sadayappan. Selective reservation strategies for backfill job scheduling, Proc. th Workshop on Job Scheduling Strategies for Parallel Processing, Edinburgh, Scotland, July.