Cycle Stealing under Immediate Dispatch Task Assignment

Transcription

1 Cycle Stealing under Immediate ispatc Task Assignment Mor Harcol-alter Cuiong Li Takayuki Osogami Alan Sceller-Wolf Mark S. Squillante Carnegie Mellon University Pittsburg, PA 11 (arcol, (cuiong, IM Researc ivision Tomas J. Watson Center Yorktown Heigts, NY 19 ASTRACT We consider te practical problem of task assignment in a server farm, were eac arriving job is immediately dispatced to a server in te farm. We look at te benefit of cycle stealing at te point of te dispatcer, were jobs normally destined for one macine may be routed to a different macine if it is idle. Te analysis uses a tecnique wic we refer to as dimensionality reduction via busy period transitions. Our analysis is approximate, but can be made as close to exact as desired, and is validated via simulation. Results sow tat te beneficiaries of te idle cycles can benefit unboundedly, due to an increase in teir stability region, wile te donors are only sligtly penalized. Tese results still old even wen tere is only one donor server and beneficiary servers stealing its idle cycles. Categories and Subject escriptors C.4 [Performance of Systems]: Modeling Tecniques General Terms Performance, Algoritms Keywords Cycle stealing, task assignment, load saring, server farm, distributed system, supercomputing, matrix analytic metods, starvation, unfairness. 1. INTROUCTION Tis work was supported by NSF Career Grant CCR- 177, by NSF ITR Grant ANI-19 and by Cisco Systems and Spinnaker Networks via Pittsburg igital Greenouse 1-1. Permission to make digital or ard copies of all or part of tis work for personal or classroom use is granted witout fee provided tat copies are not made or distributed for profit or commercial advantage and tat copies bear tis notice and te full citation on te first page. To copy oterwise, to republis, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SPAA, June 7 9,, San iego, California, USA. Copyrigt ACM //...$.. Server farm arcitecture Te server farm is a common arcitecture used by busy web sites, computation centers, file servers, and any oter service wic receives more requests tan can be andled by a single server. Te server farm is popular because it allows for increased computing power wile being cost-effective and easily scalable. Te server farm arcitecture is sown in Figure 1. Incoming jobs ispatcer Figure 1: Illustration of server farm arcitecture. In a server farm, eac arriving job (request) is immediately dispatced by a ig-speed front-end router (a.k.a., dispatcer) to exactly one of te servers, wic andles te job. Common dispatcers include Cisco s Local irector [] and IM s Network ispatcer [1]. Te immediate dispatcing of jobs is crucial for scalability and efficiency; it s important tat te router not become a bottleneck. Tere is typically no communication between servers, wic may not even know of eac oters existence. Te rule used by te dispatcer for assigning jobs to servers is known as te task assignment policy. Te coice of te task assignment policy as a significant effect on te performance perceived by users. esigning a distributed server system tus often comes down to coosing te best possible task assignment policy for te given model and user requirements. Wile devising new task assignment policies is easy, analyzing even te simplest policies can prove to be very difficult: Many of te long-standing open questions in distributed computing involve te performance analysis of task assignment policies. In tis paper we consider te particular model of a server farm in wic servers are omogeneous and te execution of jobs is non-preemptive (run-to-completion), i.e., te execution of a job can t be interrupted and subsequently resumed. Our model is motivated by servers at supercom-

2 puting centers, were jobs are typically run-to-completion (see Table 1). Our model is also consistent wit validated stocastic models used to study a wide range of ig-volume Web sites [1, 1], studies of scalable systems for departmental computers witin an organization [9], and telecommunication systems wit eterogeneous servers [4]. Previous work on task assignment Te analysis of task assignment policies as been te topic of many papers. elow we provide a brief overview. We limit our discussion to task assignment in non-preemptive systems wit immediate dispatc. For a more general discussion see [1] and te references terein. y far te most common task assignment policy used is Round-Robin. TeRound-Robin policy is simple, but it neiter maximizes utilization of te servers, nor minimizes mean response time. Wen te job sizes come from an exponential distribution, te best policy for minimizing mean response time is Least-Remaining-Work, were incoming jobs are sent to te server wit te least total unfinised work [4]. Tis requires knowing te size of jobs (a.k.a., service requirements, processing requirements). In te case were te sizes are not known, ten te Sortest-Queue task assignment policy were incoming jobs are immediately dispatced to te server wit te fewest number of jobs as been sown to be optimal under an exponential job size distribution and Poisson arrival process [, 9]. Wile policies like Least-Remaining-Work and Sortest Queue perform well under exponential job size distributions, tey perform poorly wen te job size distribution as iger variability. In suc cases, it as been sown analytically and empirically tat te policy far outperforms tese oter policies wit respect to minimizing mean response time [11, ]. In te policy, some servers are designated as te sort servers and oters as te long servers. Sort jobs are always sent to te sort servers and long jobs to te long servers. Te policy is also popular in practice (e.g. Cornell Teory Center) were different server macines ave different duration limitations: 1/ our, 1/ ours, 4ours, 4 ours, etc., and users must specify an estimated required service requirement for eac job [14]. Te intuition beind te policy is tat, under ig-variability workloads, it is important to isolate sort jobs from te long jobs, as aving sort jobs wait beind long jobs is very wasteful. Te policy is also popular in supermarkets and banks, were a separate queue is created for sort jobs. ven wen te job size is not known, it as been demonstrated tat a policy very similar to, knownas te TAGS policy (Task Assignment by Guessing Size) works almost as well wen job sizes ave ig variability. Like, tetags policy significantly outperforms oter policies tat do not segregate jobs by size [1]. Motivation for cycle stealing Given te extremely ig variability of job sizes under so many computer workloads [7,, 1,,,, 7], assignment is clearly preferable to oter policies. However is still clearly not optimal. One problem is tat can lead to situations were te servers are not fully utilized: five consecutive sort jobs may arrive, wit no long job, resulting in an idle long server. Tis is especially likely in common computer workloads, were tere are many sort jobs and just a few very long jobs, resulting in longer idle periods between te arrivals of long jobs. Ideally one would like a policy wic combines te variancereducing benefit of te policy wit te igutilization property of oter policies: We would like to segregate jobs by size so as to provide isolation for sort jobs, but during times wen te long job server is free, we would like to steal te long server s idle cycles and use tose to serve incoming sort jobs. Tis would bot decrease te mean response time of sort jobs, and also enlarge te stability region of te overall system. It is important, toug, tat we permit te sort jobs to use te long server only wen tat server is free, sotatwedon tstarve te long jobs. Noneteless, because jobs are not preemptible, tere will still be some penalty to a long job wic arrives to find a sort job serving at te long server. Our specific cycle stealing algoritm, called CS-Immediate-ispatc, or CS-Immed-isp for sort, will be described in Section. eneficiaries and donors Above we ve used te terms sort server to describe te server designated for sort jobs and long server to describe te server designated for long jobs, but wic can be used for new sort arrivals if idle. Our reason for talking about a sort server and a long server is to empasize te tremendous performance benefit acievable wen jobs can be segmented by size. Te analysis in tis paper, owever, applies more generally to any situation were tere is a beneficiary server and a donor server, were newly arriving beneficiary jobs may use te donor server if it is idle. Trougout, we will terefore use te terms beneficiary and donor jobs/servers. For completeness we will consider tree cases: beneficiary jobs sorter tan donor jobs; beneficiary jobs indistinguisable from donor jobs; and beneficiary jobs longer tan donor jobs. We will find tat te beneficiary jobs benefit in all tree cases. Te donor jobs suffer little, except in te case were te beneficiary jobs are muc longer tan donor jobs, causing donor jobs to sometimes get stuck waiting. ut even in tis case, it will turn out tat te penalty to te donor jobs is dominated by te benefits to te beneficiary jobs. ifficulty of analysis and new analytic approaces Cycle stealing is a very old concept, and policies based on cycle stealing ave been suggested in countless papers. However until tis year ([1, ]) te analysis of cycle stealing as eluded researcers. Tis paper provides te first analysis of cycle stealing under immediate dispatc task assignment. Our primary goal is to derive te mean response time for te beneficiaries and te mean response time for te donors. 1 Observe tat even for te simplest instance of our problem were job arrivals are Poisson and beneficiary jobs and donor jobs are drawn i.i.d. from respective exponential distributions te continuous-time Markov cain representation of te system is matematically difficult. Tis is due to te fact tat te state space (number beneficiary jobs, number donor jobs) grows infinitely in two dimensions (). Wile truncation of te Markov cain is possible, te errors introduced by ignor- 1 Te response time is defined as te time from wen te job arrives until it leaves te system.

3 Name Location #Servers Server Macine olas [1] MIT Lab for Computer Science -processor Ultra HPC SMP Pleiades [] MIT Lab for Computer Science 7 4-processor Alpa 114 macine J9 distributed server NASA Ames Researc Lab 4 -processor Cray J9 macine J9 distributed server [1] Pittsburg Supercomputing Center -processor Cray J9 macine C9 distributed server [] NASA Ames Researc Lab 1-processor Cray C9 macine Table 1: xamples of server farms described by te arcitectural model of tis paper. Observe tat eac server macine is a multi-processor macine. Te scedulers used are Load-Leveler, LSF, PS, or NQS. Tese scedulers typically only support run-to-completion (non-preemptive) witin a server macine because (i) timesaring witin a multiprocessor server is not supported [Parsons, Sevcik 97], and (ii) te uge memory requirements of tese parallel jobs make timesaring too expensive [Feitelson et. al. 97]. Typically in suc settings users submit an upper bound on teir job s CPU requirement, beyond wic point te job is killed. ing portions of te state space (infinite in ) can be quite significant, especially at iger loads. Tus truncation is not sufficiently accurate nor robust for our purposes. Our approac consists of several ideas. First, we define our CS-Immed-isp algoritm in a way tat will allow te decomposition of te system into two processes: te beneficiary server process and te donor server process. We can solve te donor server process exactly, providing a closedform expression for te Laplace transform of response time of donor jobs. We can also derive all moments of te busy period duration for te donor server, were te donor server s busy period is defined as te time from wen te server becomes busy until it becomes idle again. We next analyze te beneficiary server. Normally tis would require tracking bot te number of donor and beneficiary jobs in a infinite cain. However, we sow tat we can extract all te information we need by tracking only te number of beneficiary jobs and weter te donor server is busy or not. To do tis we use a special type of transition in our Markov cain wic we call a busy period transition, and wic represents te lengt of te donor server s busy period. Suc transitions allow us to represent te beneficiary server state using a 1 infinite cain. Tis cain can be easily solved using known numerical (matrix analytic) tecniques. Wile a closed-form solution is preferable, our cain is compact enoug, and matrix analytic metods powerful enoug, tat only a couple seconds are required to generate any of te results curves in tis paper. Te only approximation in our metod wit respect to tis paper lies in te accuracy of te representation of te donor server s busy period. Tis can be made as accurate as desired, as all moments of te donor server s busy period are known. In tis paper we matc moments and verify via simulation tat tis is sufficient. To summarize, we are able to analyze tis eretofore intractable problem by (i) defining our policy so as to allow decomposition of te server states, and (ii) using busy period transitions to reduce dimensionality. Te above analysis allows for very general conditions. Te service requirements of te beneficiary and donor jobs are assumed to be drawn i.i.d. from any general distribution For =1.1 andρ =.7, truncation leads to % error, even wit states, and takes > 1 minutes to compute (ere represents te load at te beneficiary server and ρ te load at te donor server). As nears ( < is necessary for stability wen ρ =.7), te error increases indefinitely. Under job sizes more variable tan exponential, te error is greater. (wic we model using a Coxian distribution []). Te arrival process is assumed to be Poisson, but can be extended to a MAP process (Markovian Arrival Process) []. Te analysis also generalizes nicely to te case of n beneficiary servers. Te work in tis paper complements two oter papers: [1] and []. Tese oter papers also look at te question of cycle stealing, owever under different models wic are not appropriate for server farms. In [1], tere is a central queue at te dispatcer and no queueing at te servers. Tis model lacks practicality for server farms, but is interesting teoretically since it leads to a muc larger stability region as compared to immediate dispatc, since donors can elp wit all beneficiary jobs, not just new arrivals. Te analysis of te central queue policy does not lend itself to decomposition of te servers (as in tis paper). Terefore te analysis in [1] requires making certain independence approximations. In [], te model differs still furter. Similarly to [1], te donors can elp wit all beneficiary jobs, not just new arrivals. In addition, te service is preemptive, wic means tat te donor server quits work on a beneficiary job wen new donors arrive. Te preemptive service model greatly simplifies te analysis, wic enables te analysis of additional parameters like switcing costs and tresolds. In bot [] and [1] it does not appear possible to analyze te case of multiple beneficiary servers, as in tis paper. Outline Section presents te CS-Immed-isp algoritm. Sections troug discuss te case of a single beneficiary server and a single donor server, including: te analysis of CS-Immed-isp (Section ); stability criteria (Section 4); and results sowing te gain to beneficiary jobs and te penalty to donor jobs (Section ), as a function of load, job sizes, and variability in te job size distributions. Section discusses analysis and results for te case of multiple beneficiary servers. In Section 7 we validate our analysis tecnique against simulation and limiting cases. Summary of Results Our analysis leads to te following conclusions: We find tat immediate dispatc wit cycle stealing vastly Coxian distributions are a subset of pase-type distributions. Pase-type distributions are commonly used for representing general distributions as a combination of exponential distributions of different rates, wic allows te general distribution to be modeled in a Markov cain.

4 ispatcer eneficiary server onor server eneficiary jobs first ceck if donor is idle. If so, go tere. lse come ere. onor jobs always dispatced ere. Figure : CS-Immediate-ispatc algoritm. improves te performance of te beneficiaries. y increasing te stability region for te system, CS-Immed-isp results in finite response time even wen te beneficiary load exceeds 1. We find tat te donor jobs do not suffer muc by aving teir idle cycles stolen. Wen te beneficiary jobs are smaller on average tan te donors, te suffering of te donors is truly negligible. ven wen te beneficiary jobs are ten times larger on average tan te donors, te donor jobs response times increase by only a small factor, and tis increase is dwarfed by te infinite improvement possible for beneficiary jobs. Te performance of te beneficiary jobs is surprisingly insensitive to te variability of te donor job size distribution, wit noticeable impact only in te region were te beneficiary load exceeds 1. Te performance of donors is negatively impacted by iger variability in beneficiary jobs. We find, importantly, tat te benefit to beneficiary jobs remains substantial and te penalty to donor jobs remains relatively low, even wen tere are beneficiary servers. Tis underscores te power of cycle stealing and is important news for many applications outside of server farms, e.g., te saring of unused bandwidt by multiple flows in Weigted Fair Queueing (WFQ) routers. Interestingly, we see tat te cange in going from i to i+1 beneficiary servers is strongly nonlinear. Tere is a big drop-off in going from 1 to beneficiary servers, but a muc smaller cange in going fromtobeneficiaryservers.. ALGORITHM AN NOTATION Te natural immediate-dispatc cycle stealing algoritm migt look someting like tis: onor jobs are immediately dispatced to te donor server. eneficiary jobs are immediately dispatced to te beneficiary server, unless te beneficiary server is busy and te donor server is idle, in wic case te beneficiary job goes to te donor server. Unfortunately, tis algoritm does not appear tractable because te stocastic process defining te system cannot be decomposed: ven te analysis of only te donor server requires keeping track of bot te number of beneficiary jobs and te number of donor jobs, wic is still a -infinite cain. Fortunately, if we perturb te algoritm just sligtly, we produce a policy wic we can decompose and analyze using te metod of dimensionality reduction via busy period transitions. Our algoritm is called CS-Immed-isp (cycle stealing under immediate dispatc). Tere is a designated beneficiary job server and a designated donor job server. An arriving donor job is always dispatced to te donor job server. An arriving beneficiary job first cecks if te donor job server is idle. If so, te beneficiary job is dispatced to te donor job server (regardless of te number of jobs at te beneficiary queue). If owever te donor job server is not idle (eiter it s working on a donor job or a beneficiary job), ten te arriving beneficiary job is dispatced to te beneficiary server. Jobs at a server are serviced in FCFS order. Te CS-Immed-isp algoritm is sown in Figure. Te CS-Immed-isp algoritm is an improvement over simple, since a fraction of te arrival stream of beneficiary jobs can be offloaded to te donor server, wile only sligtly penalizing donor jobs. In te more general case (see Section ), tere may be several beneficiary servers and a single donor server. Jobs destined for beneficiary server i will first ceck if te donor server is idle and if so go tere. If not, tey will go to teir designated beneficiary server. Trougout we assume tat beneficiary (respectively, donor) jobs arrive according to a Poisson process wit rate λ (respectively, λ ). Te size, a.k.a. service requirement, of beneficiary jobs (respectively, donor jobs) is denoted by te random variable (r.v.) (respectively, )andisassumed to be represented by a Coxian distribution. Te it moment of te size of a beneficiary (respectively, donor) job is terefore [](respectively,[ i ]). i Te load at te beneficiary server (respectively, donor server) is defined to be = λ [ ](respectively,ρ = λ [ ]). In te case of multiple beneficiary servers, we ave arrival rates: λ 1, λ,..., λ n,witjobsizes 1,,..., n,respectively. We assume tat te first tree moments of te busy periods are finite, and te queues are stable.. ANALYSIS FOR TWO SRVRS Our analytic approac involves a tree-step process: analyzing response time for te donor jobs; deriving te donor server busy period; and analyzing te beneficiary server by incorporating results from te donor server busy period. Te first step is accomplised by analyzing te donor job server. Observe tat te state of te donor server is independent of te state of te beneficiary server: Te donor server receives arrivals at rate λ + λ wen it is idle, but receives arrivals at rate only λ wen it is busy. Furtermore, wen te donor server is idle, te probability tat te next arrival is a beneficiary job, as opposed to a donor job λ is λ +λ. Te simplest way to analyze suc a system is via virtual waiting time analysis. To avoid disturbing te flow of te paper, we postpone te proof of tis teorem to Appendix. Teorem 1. Let T () represent te response time of donor jobs and let e T () (s) denote it s Laplace transform. Ten: et () (s) = s + λ e (s)λ T ()i = (T () ) i = π s λ + (s)λ e (s); g ρ ρ 1 ρ [ + ] 1+ [ + []; ] ρ ρ 1 ρ [ + ] 1+ [ ] + ρ 1 ρ [ ( T ()i [ ]) ] + [ ] T ()i + [] ;

5 were π = 1 ρ. 1+ Here π represents te fraction of time tat te donor server is idle. Our second step is to derive all moments for te busy and idle periods of te donor server, for use in our analysis of te beneficiary server. Te donor server idle time is exponentially-distributed wit rate λ + λ.toderivete lengt of te busy period for te donor server, we need to distinguis between two types of busy periods: (1) A busy period made up entirely of donor jobs, wose duration is represented by te r.v. ; and () A busy period started by one beneficiary job followed by zero or more donor jobs, wose duration is represented by te r.v.. We ten ave te following Laplace transforms: g (s) = g (s + λ λ g (s)); ] (s) = g (s + λ λ g (s)). From tese transforms we compute te first tree moments of eac type of busy period as follows: [ ] = [] 1 ρ ; [ ] = [] 1 ρ ; i i i = (1 ρ ; i ) = λ( ) + (1 ρ ) (1 ρ ; ) 4 = [ ] λ (1 ρ + 1 ) (1 ρ i ) ; = λ [] ( ) + (1 ρ ) (1 ρ ) + λ + + λ [ ] (1 ρ ) 4. To obtain te moments of a general busy period for te donor server, observe tat, due to Poisson arrivals, a busy period is of type wit probability λ /(λ +λ )andof type wit probability λ /(λ + λ ). Hence, denoting by te duration of a general busy period, we ave li = λ λ + λ l i + λ λ + λ l We now construct a pase-type distribution to matc as many moments of te donor server s busy period as are of interest. Many metods exist for matcing moments to pasetype distributions: [17, 1]. We find tat simply fitting a -stage Coxian distribution to te first tree moments of te donor server s busy period works sufficiently well for our purposes. Our last step is to analyze te beneficiary server. Analysis of te beneficiary server would seem to require a -infinite Markov cain wic tracks te number of donor and beneficiary jobs. However, if we use busy period transitions, a 1-infinite cain suffices as follows: Observe tat te arrival rate at te beneficiary server is λ during times wen te donor server is busy and during times wen te donor server is idle. To represent te beneficiary server, we terefore only need to track te number of beneficiary jobs (1infinite), wile maintaining a binary state recording weter te donor server is busy. Figure (a) sows te Markov cain model for te beneficiary server, under te simplification tat job sizes are exponentially-distributed. Here te i. λ busy, busy,1 busy, µ λ + λ λ + λ λ + λ idle, busy, β 1 λ +λ β idle, idle,1 λ µ µ µ (a) λ λ λ λ λ λ 1 busy,1 1 β β β 1 β 1 β β λ +λ λ + λ λ 1 + λ β 1 β idle,1 (b) idle, β 1 β λ + λ 1 β 1 1 β busy, β idle, Figure : Markov cain for te beneficiary server. (a) Were job sizes are exponentially-distributed. (b) Were job sizes are Coxian. busy period duration for te donor server is sown as a single bold transition marked. Figure (b) is te Markov cain tat we actually solve for te beneficiary server. Here te job sizes are drawn from a -stage Coxian distribution, used to matc te first moments of te respective job size distribution. Te busy period duration for te donor server is also matced to moments by a -stage Coxian. WesolveteMarkovcaininFigure(b)fortenumber of beneficiary jobs at te beneficiary server using well-known matrix analytic metods 4. Ten via Little s Law[], we obtain te mean response time of beneficiary jobs at te beneficiary server. Aggregating beneficiary jobs at bot servers, we ten ave by Poisson-Arrivals-See-Time-Averages [4]: [ Time for beneficiary jobs ] = Pr{onorserveridle} [ ]+ Pr{onor server busy} [Time at benefic. server]. Wile te mean response time for te donor jobs is exact, te mean response time for beneficiary jobs is an approximation wic depends on te accuracy of te approximation of te busy period of te donor server. We ave matced te first tree moments of te busy period of te donor server. Greater accuracy can be acieved by matcing more moments, by using a iger degree Coxian, until in practice 4 Te matrix analytic metod [4, 19] is a compact and fast metod for solving Q (quasi-birt-deat) Markov cains wic are infinite in one dimension, were te cain repeats itself after some point. Te repeating portion is represented as powers of a generator matrix wic can be added as one adds a geometric series to produce a single matrix. very curveintispaperwicusedmatrixanalyticanalysiswas produced witin a couple seconds using te Matlab environment.

6 ρ Stability condition on ρ Immed isp ρ Figure 4: Stability region on for and CS-Immed-isp. te results are arbitrarily close to te actual quantities. 4. STAILITY CONITIONS For assignment it is required tat ρ < 1 and < 1, were ρ (respectively, ) denote te load made up of donor jobs (respectively, beneficiary jobs). For CS-Immed-isp we will see tat te region of stability is muc wider. A proof of te following teorem is in Appendix A: Teorem. Te stability condition for donor jobs is ρ < 1, and te stability condition for beneficiary jobs is te solution to: ρ < Te restriction on for and CS-Immed-isp is sown in Figure 4. Observe te advantage of cycle stealing in extending te stability region. Wen ρ is near zero, canbeasigas1... RSULTS OF TWO SRVR ANALYSIS In tis section we evaluate te results of our analysis. All figures are organized into two parts: te benefit to beneficiary jobs and te penalty to donor jobs. To evaluate tese benefits/penalties we compare wit te algoritm wic involves no cycle stealing. In all results figures, we old ρ fixed and consider te full range of stable for tree sets of mean job sizes: beneficiaries ave mean size 1 and donors ave mean size 1; beneficiaries ave mean size 1 and donors ave mean size 1; beneficiaries ave mean size 1 and donors ave mean size 1. We generated results for various beneficiary and donor jobs size distributions. ue to space limitations, we sow only a small subset of te result plots generated, but include te broader picture in our discussion below. In Figure we assume tat beneficiary job sizes are drawn from an exponential distribution and donor job sizes are drawn from a Coxian distribution wit squared coefficient of variation, C =. In Figure we consider job size distributions wit a range of C values. LookingatFigure,weseetattebenefittotebeneficiary jobs is unbounded as 1, since te mean response time for beneficiary jobs is infinite under, and only a small finite value under CS-Immed-isp (in column (a), tese finite values are wen ρ =., and 1 wen Te squared coefficient of variation, C, is defined to be te variance divided by te squared mean. ρ =.). Note: graps ave been truncated so as to fit on te page. y comparison, te penalty imposed on donor jobs by cycle stealing is always relatively small in our experiments. Tis penalty increases wit, but not greatly. Looking at Figure (row ), we see tat even wen = 1, te penalty to donor jobs is only 1% for te case were beneficiaries and donors are equal and only 1% for te case were beneficiaries are sorter tan donors. In te patological case were beneficiaries are longer tan donors, te penalty is greater. Tis is to be expected since jobs are not preemptible and a donor job may now get stuck waiting beind a beneficiary job 1 times its size. Observe tat te relative penalty to donor jobs is significantly reduced at iger donor loads (ρ =.). Figure considers te effect of increasing te job size variability of donors and of beneficiaries. Increasing donor job size variability (from C = 1 to C = to C = 49) as te expected impact on te donor jobs (an equivalent increase for CS-Immed-isp and ), owever as surprisingly little impact on te performance of te beneficiary jobs, wit te noticeable impact occurring only wen > 1. We would ave assumed te opposite: tat beneficiaries would prefer less variable donor job sizes because tat means less variable donor busy periods and more regular elp. Increasing te beneficiary job size variability as te expected impact of more penalty to te donors. Higer beneficiary variability creates iger beneficiary response times for bot algoritms (even at lower ), witout affecting eiter algoritm s stability region. In summary we see trougout tat beneficiary jobs may benefit unboundedly from cycle stealing, regardless of donor and beneficiary variability. We also seen tat te impact to donor jobs is comparatively small.. ANALYSIS AN RSULTS UNR MUL- TIPL NFICIARY SRVRS We now discuss te scenario were tere are n classes of beneficiary jobs (eac wit its own server) and a single donor server. eneficiary jobs of class i arrive wit rate λ i and ave size i (Coxian-distributed). Te beneficiary jobs of class i first ceck if te donor server is idle and if so go tere, oterwise tey go to te it beneficiary queue. Te analysis for te multi-server case is very similar to te two server case. Te system state can again be decomposed into te donor queue and te beneficiary queues. Te donor server receives P arrivals wit rate λ wen it is busy, but rate λ + i λ i wen it is idle. Following te same virtual waiting time analysis as for te two server case, we obtain te following results for te response time of a donor job: Teorem. Assuming n beneficiary job classes wit job sizes 1,..., n and arrival rates are λ 1,...,λ n, respectively, te performance of donor jobs as te following representation: P T () (s) = s + n i=1 λ (1 i i (s)) s λ + π (s); g (s)λ T ()i = i i P n ρ 1 ρ [ + i=1 P ρ i [ i ] n ] 1+ j=1 ρ j + [ ];

7 How beneficiaries gain from cycle-stealing ρ = (a) beneficiaries 1, donors 1 (b) beneficiaries 1, donors 1 (c) beneficiaries 1, donors How donors suffer from cycle-stealing ρ = (a) beneficiaries 1, donors 1 (b) beneficiaries 1, donors 1 (c) beneficiaries 1, donors 1 19 How beneficiaries gain from cycle-stealing ρ = (a) beneficiaries 1, donors 1 (b) beneficiaries 1, donors 1 (c) beneficiaries 1, donors How donors suffer from cycle-stealing ρ = (a) beneficiaries 1, donors 1 (b) beneficiaries 1, donors 1 (c) beneficiaries 1, donors 1 Figure : Results of analysis for servers, in te case were donors are drawn from Coxian distribution wit appropriate mean and C =and beneficiaries are drawn from an exponential distribution wit appropriate mean. (a) [ ]=1; [ ]=1;(b)[ ]=1; [ ]=1;(c)[ ]=1; [ ]=1. Note different scales.

8 ffect of increasing donor job size variability 1 1 C =1 C = C =49 C =49 C =1 7 4 C =1 CS I C =1 C = CS I C = C = C =1 1 1 (a) eneficiary performance C =1 CS I C =1 C = CS I C = ffect of increasing beneficiary job size variability C = C =1 C = C = (b) onor performance C =1 C = C = C = 1 (a) eneficiary performance 1. (b) onor performance T ()i = were Figure : ffect of variability in donor job size and beneficiary job size on performance. i i P n ρ 1 ρ [ + i=1 P ρ i [ i ] n ] 1+ j=1 ρ j + ρ 1 ρ [ ( T ()i [ ]) ] + [ ] π = T ()i + [] ; 1 ρ 1+ P n j=1 ρ i represents te fraction of time tat te donor ost is idle. Next we derive te busy period for te donor server. Similarlytotetwoservercaseweave: g (s) = g (s + λ λ g (s)); ^ i (s) = g i (s + λ λ g (s)). i i i i = i λ (1 ρ + 1 ) (1 ρ i ) i ; = λ i ( ) i i + (1 ρ ) (1 ρ ; ) λ + i + λ i i +. (1 ρ ) 4 Again, similarly to te -server case, we ave: li P n i=1 = λ i l i P n j=1 λ j + λ + λ P n j=1 λ + j λ i l. Finally we analyze te Markov cain for te it beneficiary queue. Tis cain is sown in Figure 7. Observe tat we only track te number of beneficiary jobs of class i, as well as tracking weter te donor server is busy or idle. Te bold transition marked will be replaced by a -stage Coxian wic matces te first moments of te donor server busy period, as computed by te above equation. Also, te stability condition is easily extended from Teorem as follows: for 1 i n, weren is te number of beneficiary servers. [ ] = [] ; i i = ; 1 ρ 1 ρ i = (1 ρ ; ) i = λ( ) + (1 ρ ) (1 ρ ; ) 4 Teorem 4. Te stability condition for te donor queue is ρ < 1, and 1+P te stability P condition for te i-t beneficiary j queue is ρ < j i j ρ, i. j In Figure we sow results for te case of 1 to beneficiary classes, were te beneficiary classes and are all i.i.d. wit exponentially-distributed job sizes wit mean 1. Tere

9 λ j λ busy,i λ i µ i µ i busy,1i λ i µ i Σ λ j + λ Σ λ j + λ Σ + idle,i idle,1i µ i busy,i idle,i Figure 7: MC for it beneficiary server sown were job sizes are exponentially-distributed. Note te similarity to Figure (a). are many interesting observations to be made (some not sown on te figure for lack of space). First, observe tat even wit beneficiaries, te response times of beneficiaries under CS-Immed-isp is still a big improvement upon, because of te enlarged stability region. Specifically, at = 1, te mean response time is infinite under, but only 41 under CS-Immed-isp for eac of te beneficiary classes (not viewable from figure). Furtermore, observe tat te cange in going from i to i + 1 beneficiaries decreases rapidly wit i: Tere is a big cange in beneficiary performance wen we increase from 1 beneficiary class to beneficiary classes. However, tere is almost no cange at allinmovingfrombeneficiaryclassestobeneficiary classes. Observe likewise tat increasing te number of beneficiary classes only sligtly effects donor performance beyond te first few beneficiaries. After tat point, te donor performance is bounded by an M/G/1 wit setup cost (tat is, an M/G/1 consisting of only donor jobs, were te first arrival to a busy period always sees a beneficiary job). 1 1 benef benef benef 1 benef i benef benef benef 1 benef i Figure : Response times for te case were tere are n beneficiary classes, all i.i.d.. ac curve sows a different value of n, wit iger values of n leading to less gain for beneficiaries and more pain for donors. In te grap we fix ρ =.. onor and beneficiary jobs are exponentially-distributed wit mean size VALIATION OF ANALYTICAL MTHO As we are proposing a new analytical sceme to arrive at near-exact calculations of waiting times in te system, it is of paramount importance tat we demonstrate te correctness of our proposed metod. In tis section, we validate te accuracy of our metod in two ways: (i) Validation against known limiting cases: We compare te output of our algoritm wit known exact results from te literature wen tese exist. ue to te complexity of our system, tis is possible only in a limited number of special cases; specifically wen te traffic intensity of one of te customer classes approaces eiter zero or te saturation point of te system. (ii) Validation against simulation: Having evaluated our approximation metods for limiting cases, we next consider intermediate loads. Computer simulation provides an effective metod for testing our analytical results over a broad range of loads, limited only by te fact tat simulation accuracy decreases as te relative traffic intensities approac saturation [, ]. 7.1 Validation against known limiting cases elow we describe two limiting cases. In bot cases we assume beneficiary jobs are exponentially distributed, and donor jobs are drawn from a Coxian distribution wit C =. Many oter distributions of job sizes were evaluated, all resulted in te same limiting beavior. Limiting Case 1: Fix,takeρ 1. Under tis case, under CS-Immed-isp it becomes increasingly difficult for te beneficiary jobs to gain access to te donor server. Tus, te mean response time for beneficiary jobs sould approac tat of an M/G/1 queue wit load. Tisisinfactte case, as sown in Figure 9 (row 1), were we fix =.9 and evaluate te response time for beneficiary jobs as ρ is set progressively closer to 1. Limiting Case : Fix ρ,take. Under tis case, under CS-Immed-isp it becomes increasingly unlikely tat a donor job will be obstructed by a sort job. Tus te mean response time for donor jobs sould approac tat of an M/G/1 queue wit load ρ. Tis is in fact te case, as sown in Figure 9 (row ), were we fix ρ =.9 and evaluate mean response time for donors as approaces. 7. Validation against simulation vent-driven simulations of CS-Immed-isp were run in C on a 7MHz Pentium III wit M RAM. We experimented wit a range of beneficiary and donor loads, mean job sizes, and variability in te job size distributions (up to C = ). ac experiment consisted of measuring mean response time over 1 arrivals wit a warmup period of, arrivals. ac experiment was repeated tirty times (using different seeds) and te average of te tirty replications was compared wit te analytically-predicted value. Almost all of our simulation results were witin 1 % of predicted analysis. In some cases te simulation numbers were a little iger tan analysis and in some cases a little lower. Figure 1 sows just a small subset of our experiments, restricted to exponential job sizes, were is eld fixed at.9 andρ is allowed to range over all stable values. Simulation replications were quite consistent at low loads and exibited ig variability at iger loads, even under an exponential job size distribution. Wen te job size distribution was Coxian (wit C = ), te variation witin te simulation results increased furter. ue to space concerns, we do not sow te Coxian plots, but we do include tose results in our discussion below. Over all our simulation experiments, we see tat te discrepancy between simulation and analysis is primarily limited to te performance of te beneficiary jobs, under ig

10 M/G/ ρ.9 1 Validation against Limiting Case M/G/ ρ M/G/ ρ.9 1 (a) beneficiaries 1 (b) beneficiaries 1 (c) beneficiaries 1 donors 1 donors 1 donors Validation against Simulation, eneficiary Performance Simulation Analysis ρ Simulation Analysis ρ Simulation Analysis ρ.7. (a) beneficiaries 1 (b) beneficiaries 1 (c) beneficiaries 1 donors 1 donors 1 donors 1 Validation against Limiting Case Validation against Simulation, onor Performance M/G/ M/G/ M/G/ Simulation Analysis Simulation Analysis Simulation Analysis (a) beneficiaries 1 (b) beneficiaries 1 (c) beneficiaries 1 donors 1 donors 1 donors 1 Figure 9: Validation of analysis against limiting cases. In row 1, =.9. As ρ 1, response times of beneficiary jobs converge to an M/G/1 wit load. In row, ρ =.9. As, response times for donor jobs converge to an M/G/1 wit load ρ. load, in te case were beneficiaries are sorter tan te donors. Tis can be explained as follows: Te only approximation in our analysis of CS-Immed-isp stems from matcing only te first moments of te lengt of te busy periods. Tese are busy periods consisting of donor jobs, or primarily donor jobs. Tere is variability in te lengt of tese busy periods, wic becomes more pronounced wen donor jobs are very long and loads are very ig. To fully capture te effect of tese busy periods, we will need to matc more moments. Using more sopisticated simulation tecniques to ameliorate te variability in simulation results caused by te ig traffic intensity will likely elp as well. Over all te simulation experiments tat we ran, te difference between analysis and simulation was almost always witin 1 %, wit iger differences occurring only at ig traffic intensity. Te max difference was under 1%. It is wort pointing out tat for eac grap in Figure 1, te simulation portion required close to an our to generate, wereas te analysis portion was computed in a second.. CONCLUSION Tis paper presents te first analysis of task assignment wit immediate dispatc and cycle stealing. It is also te first to quantify te effect of multiple beneficiaries stealing from a single donor server. Our findings are tat immediate dispatc wit cycle stealing vastly improves te performance of te beneficiaries, by increasing te stability region for te system. Furtermore, te gains obtained by te beneficiaries are surprisingly insensitive to variability in donor beavior, specifically te variability in te donor service requirements. We also find ρ ρ ρ.7. (a) beneficiaries 1 (b) beneficiaries 1 (c) beneficiaries 1 donors 1 donors 1 donors 1 Figure 1: Validation of analysis against simulation. Trougout we fix =.9 and vary ρ. tat te donor jobs do not suffer muc by aving teir idle cycles stolen. Tese same findings persist in te case of multiple beneficiary servers stealing from a single donor server. Te reduction in te amount of benefit as more beneficiary servers are added is, interestingly, igly non-linear, wit most of te reduction coming from te first additional beneficiary. Te intuition beind tis observation follows from te stability analysis of te system. Te paper also contributes an analytical metod: First, te task assignment policy is defined so as to allow decomposition of te servers. Second, te paper uses busy period transitions to reduce te dimensionality of te beneficiary server from a -infinite Markov cain (intractable) to a 1-infinite Markov cain (tractable). Te only approximation in tis metod is te approximation of te busy period duration by its first moments (were more moments can be used to acieve te desired level of accuracy). We ope tat tis approac extends to te analysis of oter systems problems. 9. RFRNCS [1] Te PSC s Cray J9 s. ttp:// [] Supercomputing at te NAS facility. ttp:// [] S. Asmussen. Queueing simulation in eavy traffic. Matematics of Operations Researc, 17(1), 199. [4] F. onomi and A. Kumar. Adaptive optimal load balancing in a nonomogeneous multiserver system wit a central job sceduler. I Transactions on Computers, 9(1):1 1, October 199.

11 [] Cisco Systems Local irector. []. Cox. A use of complex probabilities in te teory of stocastic processes. Proceedings of Cambridge Pilosopical Society, 1:1 19, 19. [7] M.. Crovella and A. estavros. Self-similarity in World Wide Web traffic: vidence and possible causes. I/ACM Transactions on Networking, (): 4, ecember [] M..Crovella,M.S.Taqqu,andA.estavros. Heavy-tailed probability distributions in te world wide web. In A Practical Guide To Heavy Tails, capter 1, pages 1. Capman & Hall, New York, 199. [9] A. premides, P. Varaiya, and J. Walrand. A simple dynamic routing problem. I Transactions on Automatic Control, AC-(4):9 9, 19. [1] M. Harcol-alter. Task assignment wit unknown duration. Journal of te ACM, 49(),. [11] M. Harcol-alter, M. Crovella, and C. Murta. On coosing a task assignment policy for a distributed server system. I Journal of Parallel and istributed Computing, 9:4, [1] M. Harcol-alter and A. owney. xploiting process lifetime distributions for dynamic load balancing. ACM Transactions on Computer Systems, 1(), [1] M. Harcol-alter, C. Li, T. Osogami, A. Sceller-Wolf, and M. Squillante. Task assignment wit cycle stealing under central queue. In Proceedings of rd International Conference on istributed Computing Systems, May. [14] S. G. Hotovy. Workload evolution on te Cornell Teory Center IM SP. In Job Sceduling Strategies for Parallel Processing,. G. Feitelson and L. Rudolp (eds.), pages 7 4. Springer-Verlag, 199. Lecture Notes in Computer Science Vol. 11. [1] G. Hunt, G. Goldszmidt, R. King, and R. Mukerjee. Network dispatcer: A connection router for scalable internet services. In Proceedings of te 7t International WWW Conference, April 199. [1] V. S. Iyengar, L. H. Trevillyan, and P. ose. Representative traces for processor models wit infinite cace. In Proceedings of te International Symposium on Hig-Performance Computer Arcitecture (HPCA), pages 7, February 199. [17] M. A. Jonson and M. F. Taaffe. An investigation of pase-distribution moment-matcing algoritms for use in queueing models. Queueing Systems, :19 147, [1] A. Lang and J. L. Artur. Parameter approximation for pase-type distributions. In S. R. Cakravarty and A. S. Alfa, editors, Matrix-Analytic Metods in Stocastic Models, pages 11. Marcel ekker, [19] G. Latouce and V. Ramaswami. Introduction to Matrix Analytic Metods in Stocastic Modeling. ASA-SIAM, Piladelpia, [] C. Leiserson. Te Pleiades alpa cluster at M.I.T.. ocumentation at: //ttp://bonanza.lcs.mit.edu/, 199. [1] C. Leiserson. Te olas supercomputing project at M.I.T.. ocumentation available at: ttp://xolas.lcs.mit.edu, 199. [] W.. Leland and T. J. Ott. Load-balancing euristics and process beavior. In Proceedings of Performance and ACM Sigmetrics, pages 4 9, 19. [] J.. C. Little. A proof of te queuing formula L = λw. Operations Researc, 9: 7, 191. [4] M. F. Neuts. Matrix-Geometric Solutions in Stocastic Models. Jons Hopkins University Press, 191. [] M. F. Neuts. Structured Stocastic Matrices of M/G/1 Type and Teir Applications. Marcel ekker, 199. [] T. Osogami, M. Harcol-alter, and A. Sceller-Wolf. Analysis of cycle stealing wit switcing cost. In ACM Sigmetrics Conference on Measurement and Modeling of Computer Systems, June. [7] V. Paxson and S. Floyd. Wide-area traffic: Te failure of Poisson modeling. I/ACM Transactions on Networking, pages 44, June 199. []. L. Peterson and.. Adams. Fractal patterns in AS I/O traffic. In CMG Proceedings, ecember 199. [9] K. W. Ross and.. Yao. Optimal load balancing and sceduling in a distributed computer system. Journal of te ACM, ():7 9, July []. Scroeder and M. Harcol-alter. valuation of task assignment policies for supercomputing servers: Te case for load unbalancing and fairness. In Proceedings of HPC, pages 11 19,. [1] M. S. Squillante,.. Yao, and L. Zang. Web traffic modeling and web server performance analysis. In Proceedings of te I Conference on ecision and Control, ecember [] W. Witt. Planning queueing simulations. Management Science, (11), 199. [] W. Winston. Optimality of te sortest line discipline. Journal of Applied Probability, 14:11 19, [4] R. W. Wolff. Stocastic Modeling and te Teory of Queues. Prentice Hall, 199. APPNI A. PROOF OF THORM Proof. Let ρ (respectively, ρ ) denote te load at te donor server (respectively, beneficiary server). ot tese quantities must clearly be < 1. We can deduce ρ from te following equation: ρ + ρ ρ = (1 ρ )+ρ = ρ =. 1+ Te first equality follows from te PASTA (Poisson arrival sees time average) principle wic implies tat te fraction of beneficiary jobs tat are dispatced to te donor server is (1 ρ ). We terefore ave te constraint tat + ρ < 1 ρ < Next we deduce ρ, using te PASTA principle wic implies tat te fraction of beneficiary jobs tat are dispatced to te beneficiary server is ρ : ρ = ρ < 1,

12 or, equivalently, ρ < arrival. RIVATION OF RSPONS TIM OF ONOR JOS Let W (t) be te virtual waiting time for te donor queue at time t. Tat is, a job arriving at te donor queue at time t would wait W (t) before it starts being processed. y te PASTA (Poisson arrival sees te time average) principle, te virtual waiting time W is equal in distribution to te waiting time. Terefore, it suffices to analyze te virtual waiting time W for te derivation of te response time of donor jobs. Te following steps allow us to obtain te moments of W = lim t W (t): (i) Set up a differential equation for W (t, s), te Laplace transform of W (t). (ii) Let t ; ten, d W (t,s), because te queue reaces dt te stationary state. Now, W (s) is obtained as a function of π. (iii) valuate W (s =)toobtainπ. (iv) ifferentiate W (s) to obtain moments of W. (i) We first set up a differential equation for W (t, s). For tis purpose, we carefully examine te relationsip between W (t) andw (t + t). First, suppose W (t) t. Since te donor server is always busy between t and t + t, only donor jobs could arrive at te donor queue. Since te arrival process is Poisson wit rate λ, te probability of aving 1 arrival in time t is λ t + o( t). Any suc arrival will ave service time. Terefore, < W (t) t w/ prob. 1 λ t + o( t), W (t+ t) = W (t)+ t w/ prob. λ t + o( t), : someting else w/ prob. o( ). Next, suppose W (t) < t. Let a random variable ɛ be te fraction of time tat te donor server was idle during (t, t+ t) given tat tere were no arrivals during tis interval. Let a random variable ɛ be te fraction of time tat te donor server was busy during tis interval given tat tere was a donor arrival. Let a random variable ɛ be te fraction of time tat te donor server was busy during te interval giving tat tere was a beneficiary arrival. Ten, W (t + t) w/pr. 1 (λ >< + λ ɛ) t + o( t), W (t)+ = ɛ t w/ prob. λ t + o( t), >: ɛ t w/ prob. λ ɛ t + o( t), someting else w/ prob. o( ). Note tat ɛ, ɛ,ɛ 1. (See Figure 11) ased on te above observation, te Laplace transform W (t + t, s) ofw (t + t) is obtained as follows: W (t + t, s) [e sw (t+ t) ], Z = [e sw (t+ t) W (t) =x]dpr(w (t) x), x= = 1+(s λ + λ (s)) t W (t, s) Z t + O( t)dpr(w (t) x) x= + + λ + (s)λ s tpr(w (t) =)+o( t). t (a) t+ t no arrival t ε ε t (b) / / arrival / arrival no arrival t+ t Figure 11: Virtual waiting time: relationsip between W (t) and W (t + t), wen(a)w (t) t and (b) W (t) < t. Tus, we obtain te next formula. W (t + t, s) W (t, s) t Z t = s λ + λ (s) W (t, s)+ O(1)dPr(W (t) x) x= + + λ + (s)λ s Pr(W (t) =)+ o( t). t Letting t in te above formula, we obtain a differential equation for W (t, s). d W (t, s) dt = s λ + λ (s) W (t, s) + λ + (s)λ s Pr(W (t) =). d W (t,s) dt (ii) Let t. Ten, because te queue reaces te stationary state. Let W (s) lim t W (t, s). Ten, W (s) is obtained as a function of π =Pr(W (t) =): W (s) = s + λ (s)λ π. s λ + (s)λ (iii) Next, we will obtain π by evaluating W (s) ats = : Note tat te Laplace transform Z(s) of a probability distribution Z always as te property Z() = 1. 1= W () = 1+[]λ π. 1 [ ]λ Te second equality follows from te L Hopital s rule. Terefore, π = 1 λ[] 1+λ. [ ] (iv) Te moments of waiting time, and subsequently response time, are easily obtained by differentiating W (s) and evaluating at s =. In particular, te n-t moment of W is [W n ]= W (n) (). t