Towards Autonomic Virtual Applications in the In-VIGO System

Transcription

1 Towards Autonomic Virtual Applications in the In-VIGO System Jing Xu, Sumalatha Adabala, José A. B. Fortes Advanced Computing and Information Systems Electrical and Computer Engineering University of Florida June 14, ICAC 05 Advanced Computing and Information Systems laboratory

2 Outline Background Autonomic virtual applications Experimental evaluation Related work Summary Advanced Computing and Information Systems laboratory 2

3 Overview Challenge: highly dynamic grid resources lack performance guarantees Goal: enable applications to automatically Adapt to grid dynamics Recover from performance faults Resource failures Resource overloading Contribution: an autonomic application-centric middleware component for self-optimizing and self-healing computation Advanced Computing and Information Systems laboratory 3

4 In-VIGO System Virtual Application Manager 1 Web User Interface Virtual Application Manager 2 In-VIGO [FGCS 05] In-Virtual Information Grid Organization Provides application services on-demand Application session 3 Virtual Application Manager 3 Virtual Application (VA) An aggregation of actual tools into a logical session Appears differently from the actual applications Virtual Application Manager (VAM) Manages application session Supports multiple VA instances Resource Manager jobs Grid Resources Advanced Computing and Information Systems laboratory 4

5 In-VIGO System Virtual Application Manager 1 Web User Interface Virtual Application Manager 2 Why is my VERY SHORT job taking sooooooo long? Application session 3 Virtual Application Manager 3 Virtual Application (VA) An aggregation of actual tools into a logical session Appears differently from the actual applications Virtual Application Manager (VAM) Manages application session Supports multiple VA instances Resource Manager Make Virtual Applications Autonomic jobs Build Autonomic Virtual Application Manager Grid Resources Advanced Computing and Information Systems laboratory 5

6 Autonomic Virtual Application Manager (AVAM) Global Knowledge Base Machine static and dynamic information CPU speed, memory size CPU load, free memory size Job Execution History Input parameters Resource usage Global Scheduler Assists the AVAM to discover and allocate resources Machine Monitor Updates dynamic status to Global Knowledge Base Web Portal VAM Grid Resources In-VIGO Global Resource Scheduler Manager Global Knowledge Base Machine Monitor Advanced Computing and Information Systems laboratory 6

7 Autonomic Virtual Application Manager (AVAM) Autonomic Manager Resource Coordinator Job Controller Per-AVAM Knowledge Base Web Portal VAM In-VIGO Job Sensor Updates job status to the AVAM Knowledge Base CPU time, elapsed time, CPU percentage, maximum memory usage Job Sensor Autonomic Manager Local Knowledge Base Grid Resources Global Resource Scheduler Manager Global Knowledge Base Machine Monitor Advanced Computing and Information Systems laboratory 7

8 Autonomic Manager Autonomic Manager Inputs Monitor Analyze Execute Local Knowledge Base Predict Evaluate Assign predicted resource usage preferred machines query history records query candidate list chosen machine Global Knowledge Base Global Scheduler Job Controller Resource Coordinator Predict Predicts resource usage with given inputs Local learning algorithm Evaluate Estimates runtimes on the candidate machines Sorts the candidate machines Assign Allocates the best candidate machine Records the submission information Advanced Computing and Information Systems laboratory 8

9 Autonomic Manager Autonomic Manager Monitor Local Analyze Knowledge Base reschedule decision Execute Predict Evaluate Assign Global Knowledge Base Global Scheduler Job Controller Resource Coordinator Monitor Checks each job status periodically Collects data to Global Knowledge Base Analyze Estimates remaining execution time Decides control actions Execute Reallocates better resource Control actions: reschedule, stop Advanced Computing and Information Systems laboratory 9

10 Implementation Prototype Implemented in the context of In-VIGO system Evaluated for CPU-intensive jobs with relatively short execution times Learning algorithm [HPDC 99] Nearest-neighbor learning algorithm Distance metric: Euclidean distance of two sets of input values Monitoring Machine monitor: gets CPU load, memory usage from /proc file system Job sensor: gets process CPU time, elapsed time, CPU percentage etc. using the ps utility Advanced Computing and Information Systems laboratory 10

11 Implementation Evaluate CPUCycles ExecutionT imei = + CPUSpeed i ( 1 Loadi (i=1 M candidate machine) ) Monitor Analyze Execute Autonomic Manager Knowledge Base Predict Evaluate Assign Analyze RemainingTime k = PredictedCPUCycles CurrentCPUTime CPUSpeedm CPUPercentage k (m: execution machine, k=1 K jobs running in a VAM session) Advanced Computing and Information Systems laboratory 11

12 Implementation Job failure Machine or network fails: job sensor fails to update job status Controller sets an alarm and reschedules the job upon consecutive alarms Job hanging Job looks normal, but cannot proceed An upper threshold is set to detect job hanging Query overhead Cache job historical data in AVAM s Knowledge Base Reuse initial candidate machine list for rescheduling to avoid the global scheduler becoming bottleneck Advanced Computing and Information Systems laboratory 12

13 Nearest-Neighbor Learning Based Prediction Benchmark: TunProb Numerical Calculation of the Transmission Probability for One- Dimensional Electron Tunneling Parameters: minimum and maximum energies and number of energy steps are generated randomly Prediction error Prediction error PredictedCPUTime-CPUTime prediction error = CPUTime Drops below 15% after a hundred runs 225% 200% 175% 150% 125% 100% 75% 50% 25% 0% Run sequence number Advanced Computing and Information Systems laboratory 13

14 Scheduling Three scheduling strategies Round robin Best candidate w/o rescheduling Uses dynamic resource information for resource allocation Best candidate w/ rescheduling Also monitors job s progress and reschedules it if necessary Non-dedicated resource Background load: CPU-intensive processes Inter-arrival times and runtimes follow Poisson process Four load environments from unloaded to heavily loaded Advanced Computing and Information Systems laboratory 14 Load unloaded lightly loaded medium loaded heavily loaded Average and standard deviation of load in four loading scenarios

15 Scheduling Runtime (s) Benchmark: TunProb Fifty runs with random inputs are submitted in sequence Performance Better candidate promises better execution time Rescheduling adapts to changing environment and delivers even better performance Round-robin Best Candidate w/o Rescheduling Best Candidate w/ Rescheduling unloaded lightly loaded medium loaded heavily loaded Average execution times in four loading scenarios with three strategies Percentage of jobs meeting deadline 120% 100% 80% 60% 40% 20% 0% unloaded Round-robin Best Candidate w/o Rescheduling Best Candidate w/ Rescheduling lightly loaded medium loaded heavy loaded The percentage of jobs meeting deadline in four loading scenarios with three strategies Advanced Computing and Information Systems laboratory 15

16 Rescheduling A TunProb job is submitted to an unloaded machine Case 1: Different amount of load was introduced Case 2: Equal load was introduced at different time Runtime (seconds) application duration2 restart overhead application duration1 Runtime (seconds) application duration2 restart overhead application duration Load Load introduction time (seconds) Case 1: The more load, the more performance improvement Case 2: The later load introduced, the less benefit Advanced Computing and Information Systems laboratory 16

17 Related Work AppLes Users provide performance model AVAM uses learning algorithm to predict Condor Users specify the resource requirements to match appropriate resources AVAM selects resources based on prediction and evaluation Condor, GrADS Checkpointing and migration of long-running jobs AVAM resubmits short jobs Advanced Computing and Information Systems laboratory 17

18 Summary Autonomic component (AVAM) in a grid computing system (In-VIGO) Global/local knowledge bases Monitors resource and execution status Learning based prediction General performance models of CPUintensive applications Performance improvement under dynamic loading conditions Advanced Computing and Information Systems laboratory 18

19 Acknowledgements NSF Middleware Initiative NSF Research Resources IBM Shared University Research VMware In-VIGO team at UFL In-VIGO prototype can be accessed from courtesy accounts available. Advanced Computing and Information Systems laboratory 19

20 References [FGCS 05] S. Adabala, V. Chadha, P. Chawla, R. Figueiredo, J. Fortes, I. Krsul, A. Matsunaga, M. Tsugawa, J. Zhang, M. Zhao, L. Zhu, and X. Zhu. From Virtualized Resources to Virtual Computing Grids: The In-VIGO System, Future Generation Computing Systems, Vol. 21/6, pages [ICAC 04] Steve R. White, James E. Hanson, Ian Whalley, David M. Chess, Jeffrey O. Kephart, An Architectural Approach to Autonomic Computing. Proceedings of ICAC, May, [HPDC 99] N. Kapadia, J. A. B. Fortes and Carla E. Brodley, Predictive Application-Performance Modeling in a Computational Grid Environment, Proceedings of HPDC, August [CCG 03] Vadhiyar, S.Dongarra, A Performance Oriented Migration Framework for the Grid, Proceedings of Cluster Computing and the Grid, Tokyo, Japan, May, Advanced Computing and Information Systems laboratory 20