Q-LEARNING BASED PROPORTIONAL CONTROLLER FOR FED-BATCH YEAST FERMENTATION

Transcription

1 -LEARNING BAED PRPRTINAL CNTRLLER FR FED-BATCH YEAT FERMENTATIN H..E. Chuo, M.K. Tan, H.J. Tha and K.T.K. Teo chool of Engineering and Inforation Technology, Universiti Malaysia abah, Jalan UM, 884 Kota Kinabalu, abah, MALAYIA. Eail: Tel: ext. 36/3319, Fax: ABTRACT This paper explores the applicability of -learning to tune proportional controller in a fedbatch yeast ferentation process to achieve ultiobjectives optiization. Noral proportional controller has faced difficulties in optiizing nonlinear yeast ferentation due to its inherent control characteristics and poor perforance in unstable syste. -learning, on the other hand, is well-known for its interactive properties with the process environent and is suitable for the learning of syste dynaicity. Therefore, the utilization and perforance of -learning to seek for the optial gain for proportional controller is studied in this paper. Meanwhile, the perforance of -learning based proportional controller under the process disturbance is also tested to copare with noinal exponential feeding and scheduled proportional controller, which are coon feeding strategy in fed-batch yeast ferentation process. INTRDUCTIN In plant-wide cheical process control, the three ode feedback controller is the ost coonly applied ethod due to its siple operation. However, though it has satisfactory guessing and control on process gains based on error, for a fed-batch ferentation bioprocess which product concentration precision is highly required, a noral PID controller is bound by the sensors proble to aintain the control action (ets et al., ). In recent years, the research focus on artificial intelligence in ferentation optiization has brought about breakthroughs in classic control ethods. For exaple, differential evolution technique (Yüzgeç, 1; Kapadi and Gudi, 4) has been applied for the stochastic search of product optiization. This approach requires the knowledge of effective optial range in order to iniize the coputation tie and avoid trap in local optiization. Neural-network based odel training for odel predictive control has faced challenges in training and apping the large nuber of data until sall-error odel of satisfactory can be obtained ( ). The -learning algorith, well-known for its explorative and interactive properties with the process environent, is a worth-trying alternative to handle the optiization of the dynaic ferentation. In this paper, the applicability and the developent of -learning to tune the proportional controller in a fed-batch yeast ferentation is of the ajor interest for the ultiobjectives optiization. The robustness of -learning based proportional controller under the influence of disturbance will also be discussed in this paper, in coparison with noinal exponential feeding (Chuo et al. 11; Teo et al., 1) and scheduled-gain proportional control under the sae process basis. 119

2 IN The dynaic in the yeast ferentation plant is represented by the suarized and rearranged aterial balances using the paraeters in Yüzgeç (1): X, X E, X M (1) () E E X, X E, X (3) (4) X, X E, X L * (5) where dc, dc X, dc E, dc, and dv denotes the change of concentration of substrate, yeast, ethanol, oxygen and bioreactor volue respectively. Assue Monod growth kinetics, the substrate consuption rate, which can go through either oxidative,,x or ferentative,,red consuption. The rate of oxidation of substrate in the cell,,x is deterined by the sallest rates at which glucose and oxygen are taken up by the cells. n the other hand, the rate of oxidation of ethanol, E,X in cell is deterined by the sallest of the rates at which ethanol and liited oxygen are taken up by the cells. pecific growth rate, µ and respiratory quotient, R expressed in ters of,,x and E,X are shown as below., LIM E, X E, UP C Ce.38.1 C 5, X in C.359 e, up in 1.136,li,li e t C C C / e, ox (6) (7) (8) (9) (1), ox e, ox (11) 113

3 , ox e, ox (1), ox e, ox METHDLGY In a -learning algorith, a learning agent actively interacts with the dynaic environent. The agent decides the best action that causes transition of state to the environent and the latter responds in a new state to the agent. A desirable action is taken based on the reward function which specifies the overall objective of the learning. A reward function assigns rewards or penalties depending on the incorporated preference indices that tell the agent the best way to achieve the goal (yafiie et al., 8). The difficulty of designing the reward function by putting it in the shoe of different dynaic situation lies in the deterination of these preference indices and the effective range. Multistep action -learning Multistep action (MA) -learning (choknecht and Riediller, 3) considers the inborn delay nature of the ferentation syste to react to the substrate feeding. In MA, the algorith will look into the effect (resulting rewards) after executing a sequence of nuber of actions. In the proposed algorith, constant action is executed throughout nuber of steps before the agent decides on the next action for the next round of ultistep. The execution of ultistep actions, a cause transitions to the environent, resulting in state s. The axiu -value coes fro the execution of the best action, deterined by the reward function r, resulting in the axiu state. The -value is updated using the - learning function: t 1 t an A (13) where t (s,a ) is the -value for state-action at tie t, is the learning rate, is the discount factor. The learning rate deterines the iportance of past experience and the discount factor weighs the iportance of near ter rewards (Chuo et al., 11). -learning based proportional controller In this paper, -learning (L) based proportional controller is used to react to the gain, K P that ultiplies the feedback error, E t to the substrate feed flow rate, F, as shown in Figure

4 R set E t s t+1 s t Action Action P u t Plant y t election tore Max data Controller -Table -learning Agent C, C X, C E, C, V R Figure 1: -learning based proportional control process flow diagra. The various states at tie t and the calculated error, E t, are related to the process gain, K P by the agent using reward function r and the -learning function. The arguent axiu value deterines the optiized K P used to tune the agnitude of the error to deterine the next feed flow rate, F for the steps tie. (14) E t R set R (15) The ferentation process was run for 1 h with a total nuber of 5 actions, each for steps tie. The process step size is.1 h and is. h or 7 s. Within steps, the aount of accuulated yeast (qx) and ethanol (qe) were stored. The accuulated values instead of the final concentrations at steps were taken because the dynaicity of the syste within a period should not be represented by the final concentration alone. The difference of average specific growth rate at steps with the critical specific growth rate at.1 h -1 (qµ) was also calculated for reward calculation using the reward function: r 1qx qe 3q (16) where 1,, 3 indicates the weight of the preference index qx, qe and qµ respectively. In this case, 1 = 1, = - 1, 3 = - 1. More yeast production will result in higher r, ore ethanol production will result in lower r. Therefore, only actions with higher rewards will be chosen, which has represented the ultiobjectives optiization. The K P with the highest -value will be chosen as the gain for the L based proportional controller at each step. REULT AND DICUIN For the noinal case, the initial concentration of glucose (C s ), yeast (C x ), ethanol (C e ) and oxygen (C o ) are g/l, 9. g/l, g/l and.8g/l respectively. The substrate feeding strea has a concentration of 35 g/l, initial volue of 5 l and initial feed flow rate 1 l/h. The following results showed the concentration profiles of glucose, yeast (to be axiized) and ethanol (to be iniized) for (i) noinal exponential feeding, F=1e.3t (Figure ), (ii) 113

5 scheduled-proportional control, with linear K P increent of 3 for each steps, starting at initial K P = (Figure 3), and (iii) L based proportional control (Figure 4). ubstance concentration profiles against tie graph using exponential feeding 4 45 yeast glucose ethanol Tie (h) Figure : Concentration profiles of glucose, yeast and ethanol (left) and the substrate feeding profile (right) for noinal exponential feeding. Under optiu scheduled-proportional control, the final yeast and ethanol concentration are 3.91 g/l and negligible respectively, ore robust copare to noinal exponential feeding of 8.1 g/l and.5 g/l respectively. Under the sae initial conditions and using the range of gain, K P, suggested by trial-and-error tuning scheduled- K P -learning based proportional control can adjust and seek for the optial gain. Maxiu final yeast concentration and iniu final ethanol concentration, i.e. 4.4 g/l and negligible respectively are obtained for -learning based proportional control. Figure 3: Concentration profiles of glucose, yeast and ethanol (left) and the substrate feeding profile (right) for scheduled-proportional control. 1133

6 4 ubstance concentration profiles using -Learning yeast glucose ethanol Learning suggested feeding profile against tie graph Feed flow rate Tie (h) Tie (h) Figure 4: Concentration profiles of glucose, yeast and ethanol (left) and the substrate feeding profile (right) for -learning based proportional control. Disturbance in substrate feeding concentration can happen in ferentation and overfeeding of substrate tends to trigger the overflow etabolis. In this case, the disturbance introduced in substrate concentration change is 55 g/l at 1< t cheduled-proportional controller is unable to control the increasing ethanol concentration in the case of disturbance. The final concentration of yeast and ethanol are g/l and 19. g/l respectively for scheduled-proportional controller and 4.49 g/l, negligible respectively for -learning based proportional control, as shown in figure 5. -learning is able adapt the process gain when necessary and achieve its goal of axiizing yeast and iniizing ethanol even under the disturbance, as shown in the feeding profile in Figure 5. 4 ubstance concentration profiles using -Learning yeast glucose ethanol Tie (h) Figure 5: Concentration profiles of glucose, yeast and ethanol (left) and the substrate feeding profile (right) for -learning based proportional control under disturbance of =55 g/l at 1< t CNCLUIN In this study, -learning has shown result of satisfactory in tuning proportional controller to achieve ultiobjectives optiization, copare to noinal exponential feeding and scheduled-proportional controller. Meanwhile, -learning is able to seek for optial process 1134

7 gain that can reduce the disturbance effect in the substrate feeding strea rather than increasing-gain scheduled-proportional controller. ACKNWLEDGEMENT Gratitude is expressed to the Universiti Malaysia abah and the Ministry of Higher Education, Malaysia for their financial support on this research. REFERENCE CHU, H.. E., TAN, M. K., THAM, H. J. & TE, K. T. K. 11. ptiization of fedng algorith. In: Proceedings of 18 th Regional yposiu on Cheical Engineering, 7-8 ctober, Ho Chi Minh City, Vietna. KAPADI, M. D. & GUDI, R. D. 4. ptial control of fed-batch ferentation involving ultiple feeds using differential evolution. Process Biocheistry, 39, point optiization cooperating with predictive control of a yeast ferentation process: A neural network approach. Process Engineering Applications of Artificial Intelligence, 4, CHKNECHT, R. & RIEDMILLER, M. 3. Learning to control at ultiple tie scales. In: Proceedings of ICANN 3, 6-9 June, Istanbul, Turkey. MET, I. Y., BATIN, G. P. & VAN IMPE, J. F.. Feedback stabilization of fedbatch bioreactors: non-onotonic growth kinetics. Biotechnology Progress, 18, NNLEITHER, B. & HENE, B. 7. Controlled fed-batch by tracking the axial culture capacity. Journal of Biotechnology, 13, YAFIIE,., TADE, F. & MARTINEZ, E. 8. Model-free learning control of cheical processes. In: WEBER, C., ELHAW, M. & MAYER, N. M. (eds.) Reinforceent Learning: Theory and Applications. Vienna: I-Tech Education and Publishing. TE, K. T. K., THAM, H. J. & TAN, M. K. 1. ptiization of ubstrate Feed Flow Rate for Fed-Batch Yeast Ferentation Process. In: econd International Conference on Coputational Intelligence, Modelling and iulation, 8-3 epteber, Bali, Indonesia. ÜZGEÇ, U. 1. Perforance coparison of differential evolution techniques on process. IA Transactions, 49,