The Pennsylvania State University. The Graduate School. College of Engineering MARKET-BASED MODEL PREDICTIVE CONTROL FOR SURVIVABLE

Transcription

1 The Pennsylvana State Unversty The Graduate School College of Engneerng MARKET-BASED MODEL PREDICTIVE CONTROL FOR SURVIVABLE DISTRIBUTED INFORMATION SYSTEMS: RESOURCE ALLOCATION AND ALGORITHM SELECTION A Thess n Industral Engneerng by Seokcheon Lee 2005 Seokcheon Lee Submtted n Partal Fulfllment of the Requrements for the Degree of Doctor of Phlosophy December 2005

2 The thess of Seokcheon Lee was revewed and approved* by the followng: Soundar R.T. Kumara Dstngushed Professor of Industral Engneerng Thess Co-Advsor Char of Commttee Natarajan Gautam Assocate Professor of Industral Engneerng Thess Co-Advsor M. Jeya Chandra Professor of Industral Engneerng Susan H. Xu Professor of Management Scence and Supply Chan Management Mark Greaves Senor Research Program Manager, Vulcan Inc. Specal Sgnatory Vkram Mankonda Vce Presdent, Intellgent Automaton Inc. Specal Sgnatory Rchard J. Koubek Professor of Industral Engneerng Head of the Department of Industral and Manufacturng Engneerng *Sgnatures are on fle n the Graduate School

3 ABSTRACT As modern networks can be easly exposed to varous adverse events such as malcous attacks and accdental falures, there s a need to study ther survvablty. There are several mportant trends of modern nformaton networks. They tend to be large-scale wth dstrbuted and component-based archtectures, and the dynamc nature of operatng envronments leads them to utlze alternatve algorthms. As a result, the behavor of such an nformaton network can be controlled through resource allocaton as well as algorthm selecton. We study an nformaton network that characterzes such trends. The servce provded by the network s to produce a global soluton to a gven problem, whch s an aggregate of partal solutons of ndvdual tasks. Qualty of servce of the network s determned by the value of global soluton and the tme taken for generatng global soluton. In ths thess we desgn a scalable adaptve control mechansm along the lnes of model predctve control to support the survvablty of such networks by utlzng resource allocaton and algorthm selecton. To address adaptvty we model stress envronment by quantfyng resource avalablty through sensors. We buld a mathematcal programmng model wth the resource avalablty ncorporated, whch nvokes optmal control actons as a functon of both state and stress envronment. The programmng model s then decentralzed through an aucton market. By perodcally openng the aucton market, the system can acheve desrable performance adaptve to changng stress envronment whle assurng scalablty property. We verfy the desgned control mechansm emprcally.

4 v TABLE OF CONTENTS Lst of Fgures...v Lst of Tables...x Acknowledgements...xv Chapter 1 Introducton Problem Doman Research Objectves Organzaton of the Thess...5 Chapter 2 Problem Defnton Network Confguraton Control Actons Algorthm Selecton Resource Allocaton Qualty of Servce Stress Envronment Problem Defnton...12 Chapter 3 Background Lterature Survey Centralzed Approaches Dynamc Programmng Model Predctve Control Decentralzed Approaches Market-based Control Insect-behavoral Control Learnng-based Control...20 Chapter 4 Overall Soluton Methodology Dscusson on the Surveyed Control Approaches Soluton Methodology...22 Chapter 5 Mathematcal Programmng Model Effects of Resource Allocaton Optmal Resource Allocaton Mathematcal Programmng Model Non-adaptve Programmng Model Adaptve Programmng Model...31 Chapter 6 Model Refnement...34

5 v 6.1 System Behavor under Adaptve Programmng Model Model Refnement System Behavor under Stable Adaptve Programmng Model...37 Chapter 7 Decentralzed Coordnaton Decentralzaton Algorthms Two-ter Aucton Market Desgn Mult-ter Aucton Market Desgn...44 Chapter 8 Computatonal Ecosystem Model Related Researches Model Statement Naïve Ecosystem Model Decson Process of Naïve Ecosystem Model Analyss Stable Ecosystem Model Decson Process of Stable Ecosystem Model Analyss System Behavor n Stochastc Envronments Performance Evaluaton...64 Chapter 9 Emprcal Evaluaton Expermental Desgn Expermental Condtons Control Polces Numercal Results Completon Tme of Fxed Mode Polces QoS Behavor Resource Utlzaton Stablty Adaptvty Summary of Emprcal Study...88 Chapter 10 Conclusons and Future Research Contrbutons Future Research...91 Bblography...92 Appendx A Emergent Propertes of Proportonal Allocaton Appendx B Topology Determnaton B.1 Problem Statement B.1.1 Network Topology...105

6 B.1.2 Resource Allocaton B.1.3 Problem Defnton B.2 Soluton Methodology B.2.1 Problem Formulaton B.2.2 Heurstc Soluton B.3 Emprcal Results B.3.1 Network Descrpton B.3.2 Performance Evaluaton B.3.3 Resource Reservaton Appendx C System Behavor n No Stress Envronments Appendx D System Behavor n Stress Envronments v

7 v LIST OF FIGURES Fgure 1.1: An example UltraLog network...4 Fgure 1.2: Research road map and thess organzaton...6 Fgure 2.1: An example network confguraton...8 Fgure 2.2: Control actons: algorthm selecton and resource allocaton...9 Fgure 2.3: An example value functon...10 Fgure 2.4: Stress envronment...12 Fgure 4.1: Overall control structure...23 Fgure 5.1: An example network for the effects of resource allocaton...26 Fgure 5.2: Effects of resource allocaton on resource avalablty...26 Fgure 6.1: An example network for behavor analyss...34 Fgure 6.2: Behavor of T * under adaptve programmng model...35 Fgure 6.3: Behavor of v * s under adaptve programmng model...35 Fgure 6.4: Behavor of T * under stable programmng model...37 Fgure 6.5: Behavor of v * s under stable programmng model...38 Fgure 7.1: Two-ter auctonng model...41 Fgure 7.2: Mult-ter auctonng model...45 Fgure 8.1: The archtecture of a computatonal ecosystem model...49 Fgure 8.2: Expermental ecosystem confguraton...52 Fgure 8.3: Behavor of T * under naïve ecosystem model...53 Fgure 8.4: Behavor of v * s under naïve ecosystem model...53 Fgure 8.5: Behavor of G * s under naïve ecosystem model...54 Fgure 8.6: Behavor of MRA (t)s under naïve ecosystem model...55 Fgure 8.7: Behavor of queue length under naïve ecosystem model...56 Fgure 8.8: Behavor of URA (t)s under stable ecosystem model...59

8 v Fgure 8.9: Behavor of queue length under stable ecosystem model...59 Fgure 8.10: Behavor of T * under stable ecosystem model...60 Fgure 8.11: Behavor of v * s under stable ecosystem model...60 Fgure 8.12: Behavor of G * s under stable ecosystem model...61 Fgure 8.13: Behavor of w * s under stable ecosystem model...62 Fgure 8.14: Behavor of URA (t) n stochastc envronment...63 Fgure 8.15: Behavor of T * n stochastc envronment...63 Fgure 9.1: Expermental network confguraton...66 Fgure 9.2: Resource utlzaton under F2 control polces n no stress envronments...75 Fgure 9.3: Resource utlzaton under F3 control polces n no stress envronments...75 Fgure 9.4: Resource utlzaton under F4 control polces n no stress envronments...76 Fgure 9.5: Resource utlzaton under F5 control polces n no stress envronments...76 Fgure 9.6: Resource utlzaton under F2 control polces n stress envronments...77 Fgure 9.7: Resource utlzaton under F3 control polces n stress envronments...77 Fgure 9.8: Resource utlzaton under F4 control polces n stress envronments...78 Fgure 9.9: Resource utlzaton under F5 control polces n stress envronments...78 Fgure 9.10: Behavor of T * n determnstc envronment wth no stress (Con1-1)...80 Fgure 9.11: Behavor of T * n stochastc envronment wth no stress (Con1-2)...80 Fgure 9.12: Behavor of T * n determnstc envronment wth no stress (Con2-1)...81 Fgure 9.13: Behavor of T * n stochastc envronment wth no stress (Con2-2)...81 Fgure 9.14: Behavor of T * n determnstc envronment wth no stress (Con3-1)...82 Fgure 9.15: Behavor of T * n stochastc envronment wth no stress (Con3-2)...82 Fgure 9.16: Behavor of T * n determnstc envronment wth stress (Con4-1)...84 Fgure 9.17: Behavor of T * n stochastc envronment wth stress (Con4-2)...84 Fgure 9.18: Behavor of T * n determnstc envronment wth stress (Con5-1)...85

9 x Fgure 9.19: Behavor of T * n stochastc envronment wth stress (Con5-2)...85 Fgure 9.20: Behavor of T * n determnstc envronment wth stress (Con6-1)...86 Fgure 9.21: Behavor of T * n stochastc envronment wth stress (Con6-2)...86 Fgure 9.22: Behavor of URA A7 under PCYY n determnstc envronments wth stress...87 Fgure 9.23: Behavor of URA A7 under PCYY n stochastc envronments wth stress...87 Fgure B.1: Expermental task flow structure for topology determnaton problem Fgure B.2: Expermental network topologes Fgure B.3: The effects of resource reservaton n determnstc envronment Fgure B.4: The effects of resource reservaton n stochastc envronment Fgure C.1: Behavor of v * under RCYY n determnstc envronment wth no stress (Con1-1) Fgure C.2: Behavor of v * under PCNN n determnstc envronment wth no stress (Con1-1) Fgure C.3: Behavor of v * under PCYN n determnstc envronment wth no stress (Con1-1) Fgure C.4: Behavor of v * under PCYY n determnstc envronment wth no stress (Con1-1) Fgure C.5: Behavor of v * under RCYY n stochastc envronment wth no stress (Con1-2) Fgure C.6: Behavor of v * under PCNN n stochastc envronment wth no stress (Con1-2) Fgure C.7: Behavor of v * under PCYN n stochastc envronment wth no stress (Con1-2) Fgure C.8: Behavor of v * under PCYY n stochastc envronment wth no stress (Con1-2) Fgure C.9: Behavor of v * under RCYY n determnstc envronment wth no stress (Con2-1) Fgure C.10: Behavor of v * under PCNN n determnstc envronment wth no stress (Con2-1) Fgure C.11: Behavor of v * under PCYN n determnstc envronment wth no stress (Con2-1)...119

10 x Fgure C.12: Behavor of v * under PCYY n determnstc envronment wth no stress (Con2-1) Fgure C.13: Behavor of v * under RCYY n stochastc envronment wth no stress (Con2-2) Fgure C.14: Behavor of v * under PCNN n stochastc envronment wth no stress (Con2-2) Fgure C.15: Behavor of v * under PCYN n stochastc envronment wth no stress (Con2-2) Fgure C.16: Behavor of v * under PCYY n stochastc envronment wth no stress (Con2-2) Fgure C.17: Behavor of v * under RCYY n determnstc envronment wth no stress (Con3-1) Fgure C.18: Behavor of v * under PCNN n determnstc envronment wth no stress (Con3-1) Fgure C.19: Behavor of v * under PCYN n determnstc envronment wth no stress (Con3-1) Fgure C.20: Behavor of v * under PCYY n determnstc envronment wth no stress (Con3-1) Fgure C.21: Behavor of v * under RCYY n stochastc envronment wth no stress (Con3-2) Fgure C.22: Behavor of v * under PCNN n stochastc envronment wth no stress (Con3-2) Fgure C.23: Behavor of v * under PCYN n stochastc envronment wth no stress (Con3-2) Fgure C.24: Behavor of v * under PCYY n stochastc envronment wth no stress (Con3-2) Fgure D.1: Behavor of v * under RCYY n determnstc envronment wth no stress (Con4-1) Fgure D.2: Behavor of v * under PCNN n determnstc envronment wth no stress (Con4-1) Fgure D.3: Behavor of v * under PCYN n determnstc envronment wth no stress (Con4-1) Fgure D.4: Behavor of v * under PCYY n determnstc envronment wth no stress (Con4-1)...127

11 x Fgure D.5: Behavor of v * under RCYY n stochastc envronment wth no stress (Con4-2) Fgure D.6: Behavor of v * under PCNN n stochastc envronment wth no stress (Con4-2) Fgure D.7: Behavor of v * under PCYN n stochastc envronment wth no stress (Con4-2) Fgure D.8: Behavor of v * under PCYY n stochastc envronment wth no stress (Con4-2) Fgure D.9: Behavor of v * under RCYY n determnstc envronment wth no stress (Con5-1) Fgure D.10: Behavor of v * under PCNN n determnstc envronment wth no stress (Con5-1) Fgure D.11: Behavor of v * under PCYN n determnstc envronment wth no stress (Con5-1) Fgure D.12: Behavor of v * under PCYY n determnstc envronment wth no stress (Con5-1) Fgure D.13: Behavor of v * under RCYY n stochastc envronment wth no stress (Con5-2) Fgure D.14: Behavor of v * under PCNN n stochastc envronment wth no stress (Con5-2) Fgure D.15: Behavor of v * under PCYN n stochastc envronment wth no stress (Con5-2) Fgure D.16: Behavor of v * under PCYY n stochastc envronment wth no stress (Con5-2) Fgure D.17: Behavor of v * under RCYY n determnstc envronment wth no stress (Con6-1) Fgure D.18: Behavor of v * under PCNN n determnstc envronment wth no stress (Con6-1) Fgure D.19: Behavor of v * under PCYN n determnstc envronment wth no stress (Con6-1) Fgure D.20: Behavor of v * under PCYY n determnstc envronment wth no stress (Con6-1) Fgure D.21: Behavor of v * under RCYY n stochastc envronment wth no stress (Con6-2)...136

12 x Fgure D.22: Behavor of v * under PCNN n stochastc envronment wth no stress (Con6-2) Fgure D.23: Behavor of v * under PCYN n stochastc envronment wth no stress (Con6-2) Fgure D.24: Behavor of v * under PCYY n stochastc envronment wth no stress (Con6-2)...137

13 x LIST OF TABLES Table 5.1: Effects of resource allocaton on completon tme...27 Table 6.1: The effect of stablty on performance...38 Table 8.1: Ecosystem performance evaluaton...64 Table 9.1: Expermental condtons...66 Table 9.2: Control polces used for expermentaton...67 Table 9.3: Lower bound performance of completon tme...68 Table 9.4: Completon tme of fxed mode polces...69 Table 9.5: Numercal results: Determnstc envronments wth no stress...70 Table 9.6: Numercal results: Stochastc wth no stress...71 Table 9.7: Numercal results: Determnstc wth stress...72 Table 9.8: Numercal results: Stochastc wth stress...73 Table 9.9: Summary of emprcal study...88 Table B.1: Expermental network parameters Table B.2: Expermental desgn Table B.3: Expermental results Table B.4: The effects of resource reservaton...112

14 xv ACKNOWLEDGEMENTS I would lke to express my grattude to my advsor Dr. Soundar Kumara for hs support and encouragement throughout my Ph.D. study at the Pennsylvana State Unversty. He s an enthusastc researcher who s always lookng for novel perspectves and motvatng hs students toward nnovatons. I have tred to learn hs nsghtful perspectves for the last fve years snce I started workng wth hm. I would also lke to thank my commttee members Dr. Natarajan Gautam, Dr. Jeya Chandra, Dr. Susan Xu, Dr. Mark Greaves, and Dr. Vkram Mankonda, for ther nvaluable gudance and suggestons durng my research. In fnshng my Ph.D. study, I would lke to gve specal thanks to Dr. Myun-Woo Lee (at the Seoul Natonal Unversty). The work experence wth hm n my M.S. study ndeed made a great contrbuton toward my fnshng Ph.D. study. Also, I would lke to thank my senor Dr. Yong-Han Lee who gave me meanngful advces and dscussons when I was sttng wth hm n the lab. I am ndebted to my wfe Eunsuk and my sons Seungmoon and Kyoungmoon for ther sacrfce. My wfe has been so patent and my sons have been growng up so well despte my study n the U.S. They would never know how much I feel sorry and how much I love them. Fnally, I would lke to thank my parents and parents n law, from the bottom of my heart, for ther prayers and confdence n me. I am really proud of them all.

15 1 Chapter 1 Introducton Crtcal nfrastructures n many domans are becomng ncreasngly dependent on networked systems for automaton or organzatonal ntegraton. Though such nfrastructures can mprove the effcency and effectveness, these systems can be easly exposed to varous adverse events such as malcous attacks and accdental falures [1]. Two metrcs, namely survvablty and scalablty, can be used to determne the effcency and effectveness of these systems. Survvablty s defned as the capablty of a system to fulfll ts msson, n a tmely manner, n the presence of attacks, falure, or accdents [2]. One promsng way to acheve survvablty s through adaptvty: changng the system behavor to acheve the system goal n response to the changng envronment [3]. However, unpredctable adaptaton can sometmes result n worse performance than wthout adaptaton [4]. Scalablty s defned as the ablty of a soluton to work when the sze of the problem ncreases (From Dctonary of Computng at As the sze of networked systems grows, scalablty becomes a crtcal ssue when developng practcal software systems [5]. There are several trends n buldng software systems ncludng dstrbuted computng, component technology, and adaptve software. Dstrbuted computng ams at usng computng power of machnes connected by a network. When a task requres ntensve computaton, t becomes a natural choce to acheve hgh performance. Component technology 1 utlzes the components so that developers can buld systems needed by smply defnng ther specfc roles and wrng them together [6][7]. In networks wth component-based archtecture, each component s hghly specalzed for specfc tasks. Another emergng technology s adaptve software [8][9]. Adaptve software has alternatve algorthms for the same numercal problem and a swtchng functon for selectng the best algorthm n response to envronmental changes. As modern operatng envronments are hghly dynamc, adaptve software becomes an mportant tool to acheve portable hgh performance. We study a large-scale nformaton network, whch s composed of dstrbuted software components lnked together through a task flow structure. A problem gven to the network s 1 A component s a reusable program element.

16 2 decomposed n terms of root tasks for some components and those tasks are propagated through a task flow structure to other components. As a problem can be decomposed wth respect to space, tme, or both, a component can have multple root tasks that can be consdered ndependent and dentcal n ther nature. The servce provded by the network s to produce a global soluton to a gven problem, whch s an aggregate of partal solutons of ndvdual tasks. Each component can have alternatve algorthms to process a task whch trade off processng tme and value of partal soluton. Qualty of Servce (QoS) of the network s determned by the value of global soluton and tme for generatng global soluton (.e., completon tme). For a gven topology, the network can control ts behavor by utlzng two dfferent knds of control actons: algorthm selecton and resource allocaton. Whle resource allocaton tres to effcently utlze lmted resources, alternatve algorthms can change the amount of requred resources. The resource allocaton we are addressng here, s allocatng resources of each machne to the resdng components for a gven topology. Survvablty of the network s the capablty to provde hgh QoS n the presence of accdental falures and malcous attacks. In ths thess we desgn a scalable adaptve control mechansm to support the survvablty of such networks. 1.1 Problem Doman The networks we study represent dstrbuted and component-based archtectures for provdng a soluton to a gven problem. When the sze of a problem becomes large, the sze of the network as well as the number of tasks for each component can be large. One can magne wde range of scentfc and engneerng problems that can be solved wth such archtectures. Cougaar (Cogntve Agent Archtecture: developed by DARPA (Defense Advanced Research Project Agency), s such an archtecture for buldng large-scale multagent systems. Recently, there have been efforts to combne the technologes of agents and components to mprove buldng large-scale software systems [10][11][12]. Whle component technology focuses on reusablty, agent technology focuses on processng complex tasks as a communty. Cougaar s n lne wth ths trend. In Cougaar a software system comprses of agents and an agent comprses of components (called plugns). The task flow structure n these systems s that of components as a combnaton of ntra-agent and nter-agent task flows. As the agents n Cougaar can be dstrbuted both from geographcal and nformaton content sense, the networks mplemented n Cougaar have dstrbuted and component-based archtecture.

17 3 UltraLog ( networks are mltary supply chan plannng systems mplemented n Cougaar [13][14][15][16][17]. Each agent n these networks represents an organzaton of mltary supply chan and has a set of components specalzed for each functonalty (allocaton, expanson, nventory management, etc) and class (ammunton, water, fuel, etc). The objectve of an UltraLog network s to provde an approprate logstcs plan for a gven mltary operatonal plan. A logstcs plan s a global soluton whch s an aggregate of ndvdual schedules bult by components. An operatonal plan s decomposed nto logstcs requrements of each thread for each agent, and a requrement s further decomposed nto root tasks (one task per day) for a desgnated component. As a result, a component can have hundreds of root tasks dependng on the horzon of an operaton and thousands of tasks to process as the root tasks are propagated. Fgure 1.1 shows an example of the network wth four agents resdng n two machnes. There are two communtes n the network: Supply communty (Agents 1, 2, 3) and Transport communty (Agent 4). Task Generators generate logstcs tasks by convertng the operatonal plan to logstcs requrements. Those tasks are sent to Expanders and expanded to multple tasks. In the example, Expanders n Supply communty expand a task nto two tasks, one for Allocator and one for Aggregator. Allocators n Supply communty allocate tasks to nventory assets by schedulng ther demands. As tasks are allocated to nventory assets, Inventory managers generate tasks to refll ther nventory assets consderng asset consumpton and nventory polcy. Agent 4 aggregates tasks and allocates to transport assets. An UltraLog network performs ntal plannng and contnuous replannng to cope wth logstcs plan devatons or operatonal plan changes. Intal plannng and replannng are the nstances of the current research problem. QoS of these networks s determned by the qualty of logstcs plan (value of soluton) and (plan) completon tme. These two metrcs drectly affect the performance of an operaton. As the scale of operaton ncreases there can be thousands of agents (tens of thousands of components) workng together to generate a logstcs plan. Also, as the networks are workng n a mltary envronment, they are especally vulnerable to malcous attacks and accdental falures. Now, the queston s how we can make ths system survvable to generate hgh qualty logstcs plans n a tmely manner n the presence of such adverse events.

18 4 Agent 1- Supply Task Expander Allocator Inventory Agent 3 - Supply Machne 1 Task Expander Allocator Agent 2 - Supply Task Expander Allocator Inventory Machne 2 Agent 4 - Transport Aggregator Allocator Fgure 1.1: An example UltraLog network The problem we address n ths thess can be generalzed to Grd Servce envronments. The Grd technology provdes nexpensve access to large computatonal resources across nsttutonal boundares [36]. Servces can be composed over the Internet va Web Servce technology creatng enormous opportuntes for automaton of busness processes [37]. OGSA (Open Grd Servces Archtecture: defnes a Grd system archtecture based on both the Grd and Web Servce technologes. The Grd Servce enables the ntegraton of resources and servces across dstrbuted, heterogeneous, dynamc vrtual organzatons [38]. Cost and qualty consderatons may force large number of customers to look for resources and servces va such an archtecture to solve ther own computng problems from modelng to analyss. Ubqutous computng technology embeds computers n varous objects and places for sensng and controllng envronments [39]. As ths technology s becomng more realzable t mght be ntractable to deal wth enormous amount of complex computng problems wthout the use of such an archtecture. In a Grd Servce envronment, a problem s solved by composng dstrbuted servces and resources, and, followng the trends n buldng software systems, through a task flow structure between components as a combnaton of ntra and nter servce task flows.

19 5 1.2 Research Objectves In ths research we desgn a scalable adaptve control mechansm for the nformaton networks wth dstrbuted and component-based archtecture. To acheve ths objectve, we choose the framework of Model Predctve Control (MPC). Frst, to address adaptvty, we model stress envronment by quantfyng resource avalablty through sensors. Second, we buld a mathematcal programmng model, whch predcts QoS as a functon of control actons (algorthm selecton and resource allocaton) as well as the stress envronment. Thrd, we provde an aucton market as a decentralzed coordnaton mechansm for solvng the programmng model. By perodcally openng the aucton market, the system can acheve desrable performance adaptve to changng stress envronment whle assurng scalablty property. Also, we study the behavor of a computatonal ecosystem model, whch s a smplfed network under the desgned control mechansm. Though decentralzed control s an nevtable trend n controllng networked systems, t may lead to undesrable propertes wth respect to stablty. In order to ensure a scalable adaptve control mechansm desgn, we need to address the followng three objectves: Buld a mathematcal programmng model adaptve to changng stress envronment. Desgn a decentralzed coordnaton mechansm for solvng the programmng model. Study the stablty property arsng from the decentralzed coordnaton mechansm. 1.3 Organzaton of the Thess The organzaton of the thess s as follows. In Chapter 2, we formally defne the problem n detal. We revew prevous control approaches n Chapter 3 and desgn overall soluton methodology n Chapter 4. In Chapters 5 and 6 a mathematcal programmng model s bult and n Chapter 7 the programmng model s decentralzed. Chapter 8 deals wth a computatonal ecosystem model. After showng emprcal results n Chapter 9, we dscuss mplcatons and possble extensons of ths research work n Chapter 10. Fgure 1.2 summarzes research road map and thess organzaton.

20 6 Problem Defnton Network confguraton Control actons Qualty of servce Stress envronment Chapter 2 Chapter 4 Chapter 3 Overall Soluton Methodology Model predctve control Mathematcal programmng model Decentralzaton Background Lterature Survey Centralzed approaches (DP / MPC) Decentralzed approaches (Market / Insect / Learnng) Chapter 5-6 Mathematcal Programmng Model Optmal resource allocaton Programmng model Model refnement Chapter 7 Decentralzed Coordnaton Two-ter aucton market Mult-ter aucton market Chapter 8 Computatonal Ecosystem Model Novel ecosystem model Stablty / farness / optmalty Chapter 9 Emprcal Evaluaton Performance Resource utlzaton Stablty Adaptvty Conclusons Research summary Contrbutons Future research Chapter 10 Fgure 1.2: Research road map and thess organzaton

21 7 Chapter 2 Problem Defnton In ths chapter we formally defne the problem by detalng network confguraton, control actons, qualty of servce, and stress envronment. We focus on computatonal CPU resources assumng that the system s computaton-bounded. 2.1 Network Confguraton A network s composed of a set of components A and a set of nodes (.e., machnes) N. K n denotes a set of components that resde n node n sharng the node s CPU resource. Task flow structure of the network, whch defnes precedence relatonshp between components, s an arbtrary drected acyclc graph. A problem gven to the network s decomposed n terms of root tasks for some components and those tasks are propagated through the task flow structure. Each component processes one of the tasks n ts queue (whch has the component s root tasks as well as the tasks from predecessor components) and then sends t to ts successor components. We denote the number of root tasks of component as rt. The example network n Fgure 2.1 s composed of sxteen components n fve nodes. Nne components have root tasks (e.g. rt A1 = 1000 and rt A2 = 200) and those tasks are processed and propagated through the task flow structure.

22 8 N A 6 A 5 A N1 N2 A A A 3 A 4 A 10 N4 A 11 A 2 A A N5 300 ` A 13 A 14 A 15 A 16 Fgure 2.1: An example network confguraton 2.2 Control Actons A network can utlze two dfferent knds of control actons to control ts behavor: algorthm selecton and resource allocaton. Each component processes a task by choosng one of the alternatve algorthms and utlzng allocated resources n ts resdng node. For example, Fgure 2.2 depcts that component A 7 has two alternatve algorthms whch are LP (lnear programmng) and a heurstc, and node N 4 has three resdng components (A 10, A 11, A 12 ) sharng ts CPU resource.

23 9 N A 6 A 5 A 7 LP Heurstc Algorthm Selecton 1000 N1 N2 A A 2 A A 3 A 4 A 9 A A 12 N4 CPU A 11 Resource Allocaton 500 A 13 A 14 N5 300 A 15 A 16 Fgure 2.2: Control actons: algorthm selecton and resource allocaton Algorthm Selecton A component can use one of the alternatve algorthms to process a task. Dfferent alternatves trade off CPU tme and value of soluton wth more CPU tme resultng n hgher expected soluton value. As one can fnd optmal mxed alternatves, a component has a monotoncally ncreasng convex functon, say value functon, wth CPU tme as a functon of value. We call the value n the functon as value mode that the component can select as ts decson varable. A value functon s defned wth three elements as f ( v ),v, v. Ths (mn) (max) functon ndcates that a component s expected CPU tme 2 to process a task s f (v ) wth a value mode v and v (mn) v v (max). Fgure 2.3 shows an example value functon whch s pece-wse lnear ncreasng convex functon. We assume that components cannot change the mode for a task n process. 2 The dstrbuton of CPU tme can be arbtrary though we use only expected CPU tme.

24 10 Expected CPU tme f (v ) f ( v ),v(mn), v(max) v (mn) v (max) v Fgure 2.3: An example value functon Resource Allocaton When there are multple components n a node, the network needs to control ts behavor through resource allocaton. There are several CPU schedulng algorthms for allocatng a CPU resource amongst multple threads. Among the schedulng algorthms, proportonal CPU share (PS) schedulng s known for ts smplcty, flexblty, and farness [19]. In PS schedulng threads are assgned weghts and resource shares are determned proportonal to the weghts [40]. Excess CPU tme from some threads s allocated farly to other threads. There are many PS schedulng algorthms such as Weghted Round-Robn schedulng, Lottery schedulng, and Strde schedulng [41][42][107]. We adopt PS schedulng as resource allocaton scheme because of ts generalty n addton to the benefts mentoned above. We defne the resource allocaton varable set as w = {w (t): A, t 0} n whch w (t) s a non-negatve weght of component at tme t. If total managed weght of a node n s ω n, the boundary condton for assgnng weghts over tme can be descrbed as: K n w ( t ) = ω n where w ( t ) 0. (2.1)

25 Qualty of Servce The servce provded by the network s to produce a global 3 soluton to a gven problem, whch s an aggregate of partal 4 solutons of ndvdual tasks. QoS of the network s determned by the value of global soluton and the cost of completon tme. The value of global soluton s the summaton of partal soluton values, and the cost of completon tme s determned by a cost functon CCT(T) whch s a monotoncally ncreasng functon of completon tme T. We assume that the soluton values and cost are represented n a common unt 5. Let v d be the value mode used to process d th task by component and e the number of tasks processed by component to the completon. Then, QoS s computed as: QoS e = A d = 1 v d CCT(T ). (2.2) 2.4 Stress Envronment Survvablty stresses such as accdental falures or malcous attacks, affect the system by drectly consumng resources or ndrectly nvokng defense mechansms as remedes. For example, Denal of Servce attack consumes resources drectly whle relevant defense mechansm also consumes resources n terms of resstance, recognton, and recovery [1]. A defense mechansm may move components to other nodes changng the network topology dynamcally. Moble technology provdes an nnovatve concept for managng dstrbuted systems to adapt dynamcally to changng envronments though there are techncal challenges such as securty to fulfll ts promse [43][44][45][46]. We consder both survvablty stresses and remedes as stress envronment from the vewpont of the network. It would be ntractable to address each possble stress envronment snce the space of stress envronment s hgh-dmensonal and also evolvng [18][20]. But, as we concentrate on CPU resources, a stress envronment can be regarded as a combnaton of stressors 3 We call the soluton of a gven problem as global soluton to avod the confuson wth partal soluton. 4 When a component completes a task t produces a partal soluton for the task. 5 Relatve mportance can be consdered by scalng the functons and t results n the same functon structures.

26 12 and network topology. The stressors, whch are sharng resources wth the components, may have admsson to access resources or be stealng resources wthout admsson. Fgure 2.4 descrbes a stress envronment of the example network. A stressor s consumng the resource of N 2 and the components of N 5 are movng to other nodes due to some catastrophc falure resultng n a dfferent network topology. N A 6 A 5 A 7 LP Heurstc Alternatve Algorthms 1000 N1 N2 A A 2 A A 3 A 4 A 9 A A 12 N4 CPU A 11 Resource Allocaton 500 A 13 A 14 N5 300 A 15 A 16 Fgure 2.4: Stress envronment 2.5 Problem Defnton The objectve of ths research s to develop a control mechansm to maxmze QoS by utlzng alternatve algorthms (v) and resource allocaton (w) adaptve to changng stress envronment whle ensurng scalablty as descrbed n (2.3). argmax QoS (2.3) v,w

27 The research problem has several characterstcs that help understandng the problem and developng approprate control mechansms: 13 Large-scale network: The network can be large-scale as the number of components ncreases wth the scale of the gven problem to the network. (Scalable control mechansm) Unpredctable stress envronment: To predct the behavor of stress envronment, whch s a combnaton of stresses as well as remedes, s practcally mpossble. (Practcal modelng of stress envronment) Fnte tme horzon: The tme horzon for a network to generate a global soluton s fnte and also t s one mportant element of QoS of the network. (Agle supply of control polcy) Indecomposable QoS: QoS s not decomposable to ndvdual components or tasks objectves because the completon tme s common throughout the network. (Tght coordnaton) Complex dynamcs: Components nteract wth each other and stressors through task flows as well as resource sharng. Also, these nteractons are n parallel wth control actons. (Tractable control mechansm)

28 14 Chapter 3 Background Lterature Survey In general, controllng a dynamc system can be centralzed or decentralzed. We dscuss several representatve approaches to help us develop approprate control mechansms. 3.1 Centralzed Approaches We nvestgate two centralzed control approaches: dynamc programmng (DP) and model predctve control (MPC) Dynamc Programmng Dynamc programmng (DP) was ntroduced by Bellman [65]. The basc dea s the Prncple of Optmalty, whch says that n any state along an optmal trajectory, the remanng part must consttute an optmal trajectory when that state s consdered as an ntal state. Ths prncple results n Bellman Optmalty Equaton wth dfferent forms dependng on the nature of dynamc systems. DP algorthms solve the equatons to produce reactve strateges n terms of optmal closed-loop feedback control polcy, whch s a rule specfyng optmal acton as a functon of state. Markovan decson problem (MDP), as a class of stochastc optmal control problems for dscrete-tme dynamc systems, s the smplest and the most extensvely studed problem. There are off-lne DP algorthms to desgn an optmal control polcy for MDP wth a complete and accurate model of the decson problem, such as synchronous dynamc programmng, Gauss- Sedel dynamc programmng, and asynchronous dynamc programmng [21]. As the complexty grows exponentally wth the dmenson of the state space though the prncple of optmalty reduces the complexty sgnfcantly, t takes mpractcally long tme to converge to a (near) optmal polcy. As a result, there have been some efforts to perform DP n real-tme concurrently wth the actual process of control, such as n RTDP (real-tme DP), Tral-based RTDP, and

29 15 LRTA* (learnng-real-tme A*) [21]. However, they take longer tme to converge than off-lne DP algorthms at the cost of explotaton. When there s no complete and accurate model of the decson problem, there s a need for adaptve control methods. There are two types of methods to solve these problems: ndrect and drect methods [21]. Indrect methods such as adaptve RTDP explctly model the dynamc system. System dentfcaton algorthms update parameters of the current system model and control decsons are made based on the current system model. Drect methods, on the other hand, form polces wthout usng explct system models. These methods such as Q-learnng and actorcrtc, are a form of renforcement learnng,.e., f an acton s followed by a satsfactory state, then the tendency to choose that acton s renforced [22][104]. The ndrect methods are very smple and powerful addressng the ssue of large state space dmensonalty wth enormously less computaton at each tme step compared to conventonal DP algorthms. For adaptve control methods, a central ssue s the conflct between explotaton and exploraton. Decreasng the exploraton over tme to a mnmum value would be benefcal to resolve the conflct. There are a varety of renforcement-learnng technques that work effectvely on a varety of small problems, though very few of these technques scale well to larger problems [23]. To make a real system work t would be necessary to utlze some knowledge correspondng to the system. A knowledge-free approach would not have acheved worthwhle performance wthn the fnte lfetme of the systems. In prncple, adaptve methods take longer tme to converge than nonadaptve methods at the cost of explotaton and exploraton. When the envronment s nonstatonary, t takes more convergence tme n new envronment because of the memory and system utlty becomes lower as exploraton must contnue n order to adapt to envronmental changes. There are adaptve control methods for Sem-Markov Decson Problems (SMDP) for dscrete-event dynamc systems [106]. However, as they nclude more model parameters the convergence s rather slow Model Predctve Control DP provdes optmal closed closed-loop polcy but the complexty n solvng optmalty equaton grows exponentally wth the dmenson of state space. Model Predctve Control (MPC), also called Recedng Horzon Control (RHC), s one of optmal control approaches that help overcome such a problem.

30 16 When there s a reasonably accurate process model, one way to desgn closed-loop polces on-lne s through MPC. MPC refers to a class of control algorthms that utlze an explct model to predct the future response of a system [21][24][25][26][27]. In MPC a closedloop polcy s acheved through repeated on-lne desgn of optmal open-loop polces. An openloop control polcy s a sequence of control sgnals for the gven ntal state wthout usng feedback nformaton about the system s actual behavor. For each current state, an optmal openloop polcy s desgned for fnte-tme horzon by solvng a statc optmzaton problem based on an explct process model. After applyng the frst acton from the optmal polcy, the remander of polcy s dscarded. The desgn process s repeated for the next observed state feedback. Through ths repeated procedure MPC produces a control polcy that s reactve to each current system state, a closed-loop polcy. The resultng control polcy from MPC s an optmal openloop feedback control polcy. In MPC the accuracy of process models s essental though feedback can overcome some effects of poor models. Though MPC does not gve optmal closed-loop polcy n stochastc envronments, the perodc desgn process allevates the mpacts of stochastcty. Also, ts complexty becomes sgnfcantly reduced compared to DP, and t s easy to adapt to new contexts by explctly handlng objectve functon or constrants. It however requres efforts to develop accurate process models and has scalablty problem when the mathematcal programmng model s large-scale. 3.2 Decentralzed Approaches In general, controllng dstrbuted systems by means of a central controller has several dsadvantages n terms of scalablty, robustness, and nformaton securty. The controller usually needs current knowledge about the entre system, necesstatng communcaton lnks from every part of the system to the controller. These centralzed control mechansms scale badly, due to the rapd ncrease of computaton and communcaton overheads wth ncrease n system sze. Sngle pont falure of the controller wll often lead to falure of the complete system. Centralzed controller therefore could lead to non-robust network. Informaton securty and confdentalty s another mportant barrer aganst centralzed control. The network enttes may not be wllng to reveal all the detals of them especally when they are composed of multple organzatons wth dfferent nterests.

31 17 Agent technology s a promsng approach that combnes the noton of local decsonmakng wth concerns for the dstrbuted system context [101][102]. An agent s software that s capable of flexble autonomous acton n order to meet ts desgn objectves. A multagent system (MAS) can be defned as a loosely coupled network of agents that work together to acheve system objectves beyond ndvdual capabltes or knowledge [103]. MAS lmts the complexty of the problem by parttonng the complex global problem nto a set of smpler local problems. As the decsons are decentralzed, there s no sngle-pont falure and agents do not need to reveal all the detals. One of the most nterestng phenomena of natural systems s that hghly-structured global pattern emerges over tme from the nteracton of smple components wthout any central leader [51][52][53]. There have been sgnfcant efforts to desgn MAS motvated by such selforganzng systems. Though enttes act wth a smple mechansm wthout central authorty, these systems are adaptve and desrable global performance can often be realzed. We survey three representatve decentralzed control approaches: market-based control, nsect-behavoral control, and learnng-based control Market-based Control Market-based control s descrbed as a paradgm for controllng complex systems that would otherwse be very dffcult to control, mantan, or expand [54]. Market-based control works through the nteracton of local agents n the same way as economc markets. Snce markets facltate resource allocaton n human socetes, one mght expect them to be smlarly useful n controllng dstrbuted systems such as computer networks [59]. In the sprt of Adam Smth s nvsble hand from economcs, t s reasoned that collectve behavor drven by selfnterested agents wll lead to globally desrable performance. One popular form of market-based control s aucton mechansm. Auctons have been defned as a market nsttuton wth an explct set of rules determnng resource allocaton and prces on the bass of bds from the market partcpants [55]. There s a large desgn space of aucton mechansms and subsequently a varety of auctons are avalable for desgnng MAS [118]. Enterprse and Challenger are market-based control systems proposed for mnmzng mean flow tme n dstrbuted processor allocaton problems [56][57]. When a task arrves at a processor t broadcasts a request for bds to all the agents. Agents recevng the request make

32 18 bds n terms of estmated completon tme for the job as soon as t becomes dle. After an evaluaton delay the orgnatng agent selects the best bd f there are more than one bd, otherwse t selects the next bd that comes frst. Another market-based algorthm for dstrbuted processor allocaton problems were desgned [58]. When a task arrves at a processor t s allocated ntal budget and should use the budget to mgrate from one processor to another processor and to use CPU tme n a processor. Tasks have preferences for current and ts neghborng processors based on cost, servce tme, or ther combnaton n lmt of avalable current budget. When a resource s free, tasks bd t wth a prce of current prce and some fracton of estmated remanng budget n the future. If a task s bd wns the fracton decreases, otherwse ncreases. A resource s prce s updated wth the recent transacton prce. In manufacturng envronment, a market-based control approach was desgned for schedulng truck pantng [60]. A schedulng program nteracts wth the pant booths through an aucton protocol. When a truck arrves, each booth bds for the pantng job based on setup cost, current workload, etc. Also, several market-based control algorthms were proposed for controllng calls wth respect to admsson or routng n telecommuncaton networks [61][62][64] Insect-behavoral Control There are several algorthms nspred by effectve and adaptve behavor of socal nsect colones such as ants, bees, wasps, and termtes [66]. A socal nsect colony can be consdered as a decentralzed problem solvng system through smple nteractons between them n a very flexble and robust way. An mportant and nterestng behavor of ant colones s ther foragng behavor, n partcular how ants can fnd the shortest paths between food sources and ther nest. Ants depost a substance called pheromone whle movng between food sources and nest, formng a pheromone tral. Ants tend to choose paths wth strong pheromone concentratons probablstcally. Ths smple behavor can gve rse to the emergence of shortest paths between food sources and ther nest when many paths are avalable. Inspred by such a behavor an ant algorthm was proposed to the travelng salesman problem [67]. Subsequently several ant algorthms have been desgned for controllng dynamcs systems as follows. ABC (Ant-Based Control) s a routng algorthm for mnmzng the call falure n telecommuncaton networks [68]. Each node has pheromone table that ndcates the goodness of

33 19 next nodes for each destnaton. Nodes generate ants wth random destnatons to collect global nformaton. Ants move from node to node by selectng the best next node accordng to the pheromone table and when arrvng at destnaton they update the pheromone table dependng on ther trp tme encouragng the calls to follow the shortest path. AntNet was proposed as a routng algorthm for maxmzng the throughput and mnmzng packet delay n packet-based communcaton networks [69][71]. Ths algorthm s smlar to ABC system but ants n AntNet select the next node probablstcally accordng to the routng table. AC 2 (Ant Colony Control) was desgned as a routng algorthm for maxmzng the throughput n shop floor [70]. Each arrvng job s assgned an ant and t selects next machne probablstcally accordng to pheromone tral of the job type between machnes. When a job s completed, the correspondng ant ncreases the pheromone of the job type through all the machnes t has been processed and decreases the pheromone to the machne wth dfferent job types. Ths encourages the same type of jobs to select the same machnes to mnmze the setup tme. There are addtonal nterestng behavors n ant colones wth respect to task allocaton. An ant senses the denstes of other ants and objects of nterest by smply observng the ntervals of encounters. The sensed denstes form task demand (demand ncreases wth less ants and more objects) and tasks are emergently allocated based on socal domnance of the ants n addton to the demands. Ths task allocaton behavor was appled to a moble sensor network [63]. The sensor network tres to maxmze network coverage by dynamcally allocatng the moble sensors to dfferent regons. Each moble sensor mmcs the task allocaton behavor of the ant colones. They observe the ntervals of other sensors and targets n lmted range and ther socal domnance s determned by ther actual performance. Based on the demands and socal domnances the sensors decde to stay or move to other regons. Smlar to ant algorthms, wasp algorthms were proposed nspred by task allocaton behavor of wasp colones. An ndvdual wasp has a response threshold for each zone of the nest [72][105]. Based on a wasp s threshold for a gven zone and the amount of stmulus from related task n that zone, a wasp may or may not become engaged n the task for that zone. When two wasps encounter each other, the wasp wth the hgher socal rank wll have a hgher probablty of domnatng n the nteracton. Ths task allocaton behavor was used for maxmzng the throughput n shop floor [73][74][75]. Each machne s assgned a routng wasp n charge of assgnng jobs to correspondng queue. Routng wasps have dfferent response thresholds for dfferent job types. If a machne s processng or settng up a job type, the threshold for the job type decreases and those for other job types ncrease. If a machne s dle the threshold decreases

34 20 accordng to the dle tme. Each job generates stmulus that ncreases as the watng tme ncreases. Each wasp pcks up a job probablstcally dependng on the strength of the stmulus and response threshold. If there are wasps more than one for a job the job s allocated to a wasp probablstcally dependng on the socal ranks determned by current workload Learnng-based Control Renforcement learnng can be used wthout pror knowledge of the system model. By makng agents to learn from ther experence ths method can be used n a decentralzed mode. Q- Routng was proposed as a routng algorthm for mnmzng packet delvery tme n packet-based communcaton networks [77][78]. Each node has a Q table that shows the estmates of delvery tme (ncludng watng tme n the node) through neghborng nodes for each destnaton. A packet selects a neghbor wth the lowest total delvery tme for the destnaton, and as soon as the packet s delvered to the selected node the lowest estmate from the selected node to the destnaton node s nformed to the orgnatng node. By addng ths value to the watng tme the orgnatng node estmates new delvery tme and update the Q value usng the dfference between new and old estmates wth a certan learnng rate. Also, CDRQ (Confdence-based Dual Renforcement Q) routng, a combnaton of CQ routng and DRQ routng, was desgned for packet-based communcaton networks [79][80][81]. In CQ routng each Q value has a confdence value that ndcates ts relablty. If a Q-value s not updated for a long tme, ts relablty goes down. When a packet s sent to a neghbor the neghbor sends back the confdence value as well as the lowest delvery tme for the destnaton. The confdence values of orgnatng node and the neghbor are used to determne learnng rate. In DRQ routng the system utlzes backward exploraton. When a node sends a packet to a neghbor the neghbor also updates correspondng Q value. It would be possble to bas the reward of each node such that ts objectve and the global objectve are algned. Ths concept was appled to routng n packet-based communcaton networks where each node learns to maxmze the based reward [83][84][85]. Also, a dstrbuted renforcement learnng algorthm was proposed for dstrbuted processor allocaton problems [82]. The effcency of each processor s renforced based on ts actual performance.