Adaptive Online Parameter Tuning Using Genetic Algorithms

Transcription

1 Adaptive Online Parameter Tuning Using Genetic Algorithms Paolo Dadone 1 and Hugh VanLandingham 2 1 VPI & SU, Whittemore Hall, Blacksburg, VA 24060, U.S.A., dadone@vt.edu 2 VPI & SU, Whittemore Hall, Blacksburg, VA 24060, U.S.A., hughv@vt.edu Abstract: A common practice is to design a controller by plant observations (i.e. experiments) and to optimize some of its parameters by trial-and-error. This paper proposes a genetic algorithm for the automation of the search procedure and its implementation on a programmable logic controller. The details of this implementation will be discussed along with an example one carried out for the control of a plant simulation problem introduced by Eastman Chemical Co. The advantage of such an approach consists of automating the search for a good solution. Moreover the genetic algorithm can be easily programmed in the PLC and reused for different plants with the only need of string encoding and fitness evaluation reprogramming. 1. Introduction Practical control applications are characterized by the need of efficient, easy, and robust approaches. For their ease of use PID control techniques are still widely used in the control community. Along with this, programmable logic controllers (PLCs) are often encountered in the implementation of PID controllers. Indeed PLCs offer a great flexibility in implementing PID loops. The control engineer is then left with the task of tuning the PID loops, that is, fixing set-points and PID constants. This task is generally approached in a trial-and-error way that obviously requires the use of the designer time to run and observe each experiment and consequently modify the loops parameters. The length of each experiment can be large enough to make the tuning process even more bothersome for the designer. This trial-and-error tuning can be considered as a man-carried search for the set of parameters that maximize some established performance index. In this context, different searches could be used. Gradient-based search might not be generally suitable due to the need of gradient information not necessarily available or existent (i.e. the objective function might be non-differentiable). Genetic algorithms (GAs) [1] are global random search algorithms widely employed in non-differentiable optimization problems, or in problems where the gradient of a given objective function is not readily available. They compensate the lack of gradient information through a

2 random exploration of the search space that evolves in analogy to the evolution of the species. The power of GAs consists in only needing objective function evaluations to carry out their random search. On the other hand, the task can be time consuming or lead to local extrema. In an industrial environment where models might not be readily available and controller tuning might be essential to profitable operations GAs could be used for automating the tuning process [2], thus relieving the designer from it and probably leading to a better solution. Moreover any GA implementation can be easily reused for different problems with minor changes. This paper explains the implementation issues for GA-based PLC loops tuning. Section 2 briefly reviews the operation of genetic algorithms. Section 3 discusses GAs implementation issues on a PLC. Section 4 presents an application of this approach on the control of a chemical plant simulation for Eastman Chemical Co. Finally, Section 5 concludes the paper with final remarks. 2. Genetic Algorithms Genetic algorithms [1] are search algorithms based on natural selection processes and reinforcement of the species. Because of their power and ease of implementation, genetic algorithms have been widely applied to combinatorial optimization problems, i.e., problems where complexity increases exponentially with the problem dimensions. In a basic sense, GAs consist in having a set (population) of candidate solutions (individuals) to an optimization problem that evolves at each iteration of the algorithm. The evolution of the species is obtained through a fitness function and some genetic operators such as reproduction, crossover, and mutation. The underlining idea in GAs is that the fittest individuals will survive generation after generation while also reproducing and generating offsprings that might be stronger then themselves. At the same time the weakest individuals disappear from each generation and some mutations may randomly happen. Another important characteristic of GAs is the necessity of coding; indeed the search parameters need to be coded in some alphabet (e.g., binary). Thus, to implement a GA five basic operations needs to be defined: string encoding, fitness evaluation, reproduction, crossover and mutation. The first two are problem dependent while the last three are problem independent, thus allowing for some modularity in the programming of a GA. String encoding is commonly done by using a binary alphabet and deciding a-priori the number of bits desired to represent each search parameter. The more bits implies better precision but longer run time. The precision of this coding scheme is given by: U π = max n U 2 1 where U max and U min are the upper and lower limit of the parameter range and n is the number of bits used for that particular parameter. After a coding scheme is min (1)

3 devised for each parameter, an individual can be formed from the concatenation of the strings of bits corresponding to each parameter. The fitness function is another problem dependent feature in the GA operation. It is a function that, given an individual (a set of values for the search parameters), returns its fitness. A GA algorithm will iterate trying to maximize the fitness function. The problem constraints can be embodied in the fitness through penalty (or barrier) functions that act by penalizing the individual that violates one, or more of the constraints. Upper and lower bound constraints can be directly embodied in the encoding procedure. The basic genetic algorithm can be summarized by the following steps: Step 1: Generate some initial, possibly random, individuals for a population of fixed size; Step 2: Evaluate their fitness; Step 3: Reproduce them with a probabilistic method (e.g., roulette wheel); Step 4: Execute crossover on the reproduced ones (choosing probabilistically both the crossover site and the mates ); Step 5: Execute mutation on a random bit with some low probability; Step 6: Go back to Step 2 unless some predefined convergence criterion is met. The reproduction can be executed with a roulette wheel type of selection. With this method each individual is assigned a portion of the wheel equal to the fraction of it s fitness with respect to the sum of the fitness of all the individuals in the population. In other words the more an individual contributes to the overall population fitness (sum of the individual fitness) the more it is likely to be chosen. More precisely, every individual is chosen with a probability that is the amount of contribution he gave to the overall population fitness. The process is probabilistic, thus allowing for multiple selections of the same individual and selection of low fitness individual too. Once the individuals for the next generation are determined some crossover operator needs to be applied to them. In the basic GA a simple one site crossover is executed. This corresponds to randomly finding two mates from the chosen individuals for the next generation and then randomly selecting a crossover site, that is a location between two adjacent bits in the bit strings. The bits of the first mate on the left of this location are concatenated with the bits of the second mate on the right of this crossover location and vice versa, thus obtaining two new individuals for the next generation, produced by the combination of two selected individuals of the past generation. It is important to underline that the process is completely random and it reinforces strong individuals only in the long run but not necessarily on a step by step basis. This simplistically corresponds to the fundamental theorem of genetic algorithms stating that the building blocks of good solutions spread through the population. Sometimes the spreading of good solutions can happen too fast, thus causing the risk of a premature convergence to a local maximum. One operator that somehow helps avoiding this premature convergence is the mutation operator. This corresponds to some random abrupt change in the species. A mutation probability is set (generally to a low value) such that there seldom is a mutation, that is, a random bit in an individual is inverted. The mutation helps in randomly sampling other areas of the search space that might have been unexplored and might contain local or global maxima. The probability of mutation

4 must be kept low since, otherwise, it could cause the loss of too many good individuals. A lot of features can be added to the basic GA: elitism, i.e. keeping some best individuals from each generation; two-point crossover, random template crossover, etc.; fitness sharing, i.e. fitness is decreased when many individuals are in the same neighborhood, thereby limiting the uncontrolled growth of a particular species within a population. The power of GAs is apparent: they are easy to program, they are powerful and they are specially suited for situations where no information on the derivatives is known. Moreover they require no restrictive assumptions on the search space (continuity, existence of derivatives, etc.). 3. PLC Implementation 3.1 Random Number Generation Genetic algorithms are a random search for an optimal solution and as such their operation strongly relies on the availability of random numbers. There obviously is no such thing as a random number generator available, but there are a lot of pseudo random number generators (PRNG) that can be implemented on the PLC in order to generate the necessary random numbers (for a first reading refer to Chapter 7 of [3]). The objective of a PRNG is to generate a long sequence of numbers that are independent and uniformly distributed in [0, 1]. There are a lot of different PRNGs among which the most used are the linear congruential generators (LCG). An LCG generates pseudo random numbers according to the following recursion: Z = ( a Z + c)( m) 1 mod (2) i i Every number is then normalized to 1 by dividing it by the modulus m. An LCG works by generating a sequence of numbers according to (2), unfortunately these sequence is periodic, the numbers are obviously not really random and independent and neither are they uniformly distributed. Under certain conditions an LCG will have full period, that is, it will produce a sequence of numbers with period equal to the modulus (m different numbers will be generated before starting to generate the same numbers again). The desirable full period property is guaranteed under the hypothesis of the following theorem. Theorem: The LCG defined in equation (2) has full period if and only if the following three conditions hold: (i) The only positive integer dividing both m and c is 1. (ii) If q is a prime number dividing m, then q divides a-1. (iii) If 4 divides m, then 4 divides a-1.

5 Choosing a full period LCG with high modulus makes a lot of numbers available but unfortunately it does not help in term of the statistical properties of the generator. Indeed there are a lot of commercially used PRNGs that do have bad statistical properties. A quite common and widely used generator is multiplicative (c = 0) and is characterized by a=630,360,016 and m = A LCG was implemented on the PLC with a special function subroutine. Note that one robust and statistically efficient LCG needs to be implemented only once and can be reused for different applications. 3.2 Fitness Evaluation Fitness evaluation is definitely a key issue since it corresponds to the evaluation of the objective function (i.e. the function whose maximization is sought). This particular feature is problem dependent and needs to be addressed on a case by case basis. In general it will be some steady state performance measure that relates (directly or not) to the profit of the plant. Thus, for fitness evaluation we will need only to set up a timer on the PLC that reads a certain quantity (or quantities) after a reasonable delay to reach steady state has elapsed. Once steady state is reached a special function program containing the core of the GA (core_ga) can be executed. Its first step will be to read the steady state performance measures and compute the fitness accordingly. The steady state wait process can be taken care by ladder logic as shown in Fig. 1. Note how the switch X2 starts the tuning process and the switch X3 causes intialization to occur. Whenever the timer is done counting a time P ss (i.e. the time needed to reach steady state) it will set a variable that will in turn cause the execution of the special function program containing the core_ga algorithm. As soon as the algorithm is done the other variable driving the timer (i.e. C1) is reset and set so that the timer is re-initialized and will start running again. X2 C1 C2 C2 X3 TMR P=P ss SFPGM 100 (core_ga) SFPGM 103 (Init) C2 C1 RST C1 SET Y41 Figure 1: Ladder logic for GA implementation

6 Fitness evaluation is somehow process dependent in the sense that the quantities that will form the steady state performance measure as well as the time to wait to reach steady state are problem dependent. 3.3 Encoding A simple special function subroutine for decoding and parameters loading needs to be setup. Whenever a new individual s fitness function is sought the corresponding individual (in encoded format) is passed to this subroutine. Here the individual is decoded into the corresponding values of the parameters that are loaded into the PLC loops appropriate quantities, thus implementing the controller corresponding to the next individual. The decoding is performed by bit-masking each parameter, eventually shifting it down and adding the appropriate offset. Note how the encoding process directly embodies lower and upper bounds so that if only loops setpoints are considered for optimization then stability conditions on those setpoints can be directly implemented. 3.4 Reproduction, Crossover and Mutation Whenever steady state is reached and the fitness for the incumbent individual is determined the core_ga special function program needs to determine whether that was the last individual of the population or not. In case it was then the regular GA operation need to be executed, otherwise the next individual is loaded and the timer re-initialized. Reproduction, crossover and mutation can be implemented in the core_ga special function program as it would be done in any high level programming language. Probabilities for roulette wheel selection are formed. Mates for reproduction are chosen (by sampling a pseudo-random number) and placed in a temporary memory area. By sampling another PRN a crossover site is defined and the two new individuals are formed by bit masking and placed in a temporary storage. This process is repeated until the new population is formed. Mutations are randomly executed for each individual (by sampling a PRN for each individual). The new population is then moved from the temporary storage area to the population permanent storage area and the first individual is loaded. Note how the implementation of this processes will be needed only one time since those are problem independent features of GAs. Thus there is a high degree of reusability of GAs for different problems. For ease of programming it is suggested to accumulate the fitness whenever an individual is unloaded and the next is loaded. This way a measure of the total fitness for the subsequent roulette wheel selection is already computed. Moreover a current individual counter is needed too. 3.5 Memory Requirements Some temporary PLC memory (generally V-memory) needs to be used for storage of:

7 generation under evaluation and corresponding profits; newly reproduced generation; discrete probability distribution for reproduction; current individual and current generation counter. The specific memory requirement depends on the number of bits used to represent an individual. 3.6 The core_ga Algorithm The core_ga algorithm previously mentioned can be implemented in a special function program. It can be summarized by the following algorithmic steps: (i) Read the steady state performance measures and compute the fitness of the corresponding individual storing it in the appropriate location. If this was the last individual of the generation go to step (iii) otherwise go to step (ii). (ii) Increase the current individual counter and load the next individual into the appropriate loops (i.e. call the decode special function subroutine). (iii) If the maximum number of generations has been reached stop. Otherwise, perform reproduction, crossover and mutation. Move the newly generated population from the temporary to the permanent storage, update the generation counter and exit. 4. Process Control The problem is to control a chemical plant in order to increase profit (at steadystate), while maintaining stable operating conditions. The chemical process produces a product, R, by reacting two chemicals, A and B. A second reaction also takes place and produces an undesired bi-product, S. Both reactions are exothermic and are as follows: A+B R Reaction 1 2B S Reaction 2 (3) The chemical plant, shown in Fig. 2, consists primarily of two reactors (RX1 and RX2), a distillation column, a reflux drum, and the related valves, piping and measurement devices. The process begins as chemicals A and B are fed into reactor 1, where they are allowed to mix. Next the mixture flows into reactor 2, where it further reacts. This mixture then proceeds into the distillation column. The distillation column consists of a number of trays on which the chemicals flow down. Product R is separated from chemicals A, B, and S by heating the mixture through the use of steam. Since product R has a boiling point higher than the other chemicals, it remains a liquid and flows out the bottom of the column, while the other chemicals are transformed into vapor and collected in the reflux drum. These chemicals are then cooled to the liquid state and are either sent back into the

8 distillation column or to the beginning of the process. All the flows are controlled by the corresponding valves shown in Fig. 2. The plant profit is determined by several factors: quantity and quality of the product (R), amount of B feed stocks consumed, and amount of steam used to heat the distillation column. The product sells for 0.80 $/lb if the mass fraction of B in it is less than , 0.40 $/lb if it is between and 0.005, and is not worth anything if the mass fraction of B is greater than The B feed costs $/lb and the steam costs 0.04 $/lb. A Siemens TI505 PLC was programmed through TISOFT v The interface software, TISOFT in this case, allows the user to program loops (i.e., PID controllers), special function programs, along with some additional features as well as relay ladder logic. For more references on PLCs and TISOFT see [4]. The PLC was connected to a 386/ATM coprocessor that runs a simulation of the chemical process, communicating with the PLC through the back plane using PCCOMM drivers [5]. The interfacing code [6] was developed at Eastman and added to the existing plant simulation code. Thus, the plant controller was developed and implemented on the PLC in an online fashion analogous to practical situations, with the difference that instead of an actual plant, a simulation is running and being controlled by the PLC. This does not subtract any amount of generality from the present approach which does not require any simulation or model of the plant. The overall controller is a distributed controller composed of several PI (proportional plus integral) loops for level and temperature control. Reactor 1 level is controlled with the B feed, while A feed is manipulated in order to maintain a given B-to-A ratio (r). The setpoint for reactor 1 level was chosen to be 550 lb. (within 20% of maximum level), while the ratio r is Reactor 2 level is controlled by the outflow of reactor 1; the setpoint is also 550 lb. The distillation column level is controlled (reverse acting) by the base flow. The setpoint for this controller is 400 lb. The outflow of reactor 2 also controls the distillation column level through a different loop. The setpoint for this controller is 700 lb (higher than the one of the previous loop), in order to maximize the outflow of reactor 2 while maintaining stable operating conditions. Indeed, this is a safety measure that Reflux Drum A Feed Distillate Flow Reflux Flow B Feed Rx 1 Out Coolant Coolant Rx 2 Out Distillation Column Rx 1 Rx 2 Steam Flow Product Out Figure 2: Chemical process

9 prevents the distillation column from overflowing. The reflux drum level is controlled by the distillate flow. The setpoint for this controller is 400 lb. Reactors 1 and 2 temperatures are controlled by their respective coolant flow. The setpoints for these controllers are 78 C and 93 C. The distillation column temperature is also controlled by the steam flow. The setpoint for this controller is C. The reflux is used (in another loop) to avoid excessive distillation column temperatures; it controls (reverse acting) the column temperature at 165 C. After stabilizing the plant, necessary tuning was done to increase the profit. The proportional and integral gains are not crucial design factors for steady-state profit rate maximization. Therefore, values corresponding to reasonable plant dynamics were heuristically found. Tweaking the level setpoints does not significantly affect the steady-state performance; whereas the temperature setpoints as well as the B-to-A ratio do have a significant impact on the profit. All the setpoints listed above were determined by a trial-and-error procedure with the goal of profit rate maximization. The corresponding profit is 1060 $/hr. For profit maximization three search parameters are used: r (B-to-A ratio), reactor 1 temperature setpoint and reactor 2 temperature setpoint. Four bits were used for encoding r, and five for each of the temperatures. The chromosome of an individual will thus be comprised of 14 bits. The corresponding ranges are: 1.25 to 2.00 in 0.05 increments for r, 80 C to 111 C in 1 C increments for both temperatures. An individual can be stored in a word (16 bits) where two bits are not used. The fitness function of every individual is given by the profit rate that the corresponding controller accomplishes once steady state is reached. From our experience a reasonable time to wait for steady state is 180 minutes. Therefore the fitness of an individual is defined as the average profit rate after 180 minutes of operation of the plant with controller corresponding to that individual. The characteristics of the GA that was implemented are the following: 12 individuals per generation, roulette wheel selection, one site cross over, mutation with probability 0.01 and elitism (the two best individuals are kept for the next generation). The GA was run for 60 generations yielding the following result: B-to- A ratio=1.8, reactor 1 temperature setpoint = 80 C, reactor 2 temperature setpoint = 95 C and profit rate (fitness) of $ 1108 per hour. This result is close to the one of the manual trial-and-error design. Therefore, it seems that the initial design was already very good. However, this approach results in finding optimal setpoints for the proposed controller through an automatic on-line optimization procedure. 5. Conclusions In this paper a simple approach to fine-tuning controller parameters was presented. This approach uses genetic algorithms to optimize a plant performance measure. The presented approach results in an automated on-line tweaking process without the requirement of human resources. Moreover the GA can be easily implemented on a programmable logic controller with highly reusable code. The corresponding implementation issues were discussed. A chemical process was introduced as the plant to be controlled and a conventional controller was designed through

10 observation of the system. Finally a GA was used for optimizing the profit rate of the plant. The GA was simple to implement and seemed to work well without needing too many generations to reach optimality. Building a GA and implementing it on PLCs is only required to be done once since the algorithm is problem independent. For different plants it is only needed to reprogram the string encoding and fitness evaluation. Acknowledgment We wish to thank the DuPont Co. for their support. References [1] D. E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning, Addison-Wesley Publishing Co., Inc., [2] P. Dadone and H. VanLandingham, PLC Implementation of a Genetic Algorithm for Controller Optimization, Proc. of the 3 rd Intrntnl. Conf. on Computational Intelligence and Neuroscience, Research Triangle Park, North Carolina, October [3] A. M. Law and W. D. Kelton, Simulation Modeling and Analysis, McGraw-Hill Inc., [4] SIMATIC TI505 Programming Reference Manual, Siemens, [5] SIMATIC TI /ATM Coprocessor User Manual, Siemens. [6] Intface.c documentation (internal report), Eastman Chemical, 1997.